How Allele-Frequency Choices Affect Kinship Estimates

Why the Frequency Reference Matters

A genomic relationship estimate is not calculated from genotype dosages alone. For each variant, the dosage is centered by subtracting its expected value under a chosen allele frequency. Changing that reference changes the centered genotypes, which can change the relationship estimate even when the observed genotypes are identical.

This matters especially in a structured sample. If two people come from populations with different allele frequencies, their genotypes may resemble one another because they share ancestry, not because they are close relatives. A reference that does not represent the relevant population can leave systematic shifts in the centered genotypes. Those shifts contribute to the estimated relationship.

Definition: At variant $\ell$, let $g_{i\ell}\in\{0,1,2\}$ be the number of copies of the counted allele for individual $i$, and let $p_\ell$ be the allele frequency used as the reference. The centered genotype is $z_{i\ell}=g_{i\ell}-2p_\ell$. For a set of variants, one common genomic relationship estimate is $$ \widehat{G}_{ij}= \frac{\sum_{\ell=1}^{m}(g_{i\ell}-2p_\ell)(g_{j\ell}-2p_\ell)} {\sum_{\ell=1}^{m}2p_\ell(1-p_\ell)}. $$ Here $m$ is the number of included variants. The estimate is an additive genomic relationship measure; under standard assumptions, it is on the relationship scale, which is twice the pedigree kinship-coefficient scale.

The denominator scales the sum by the expected genotype variance under the reference frequencies, since a diploid genotype under random allele sampling has variance $2p_\ell(1-p_\ell)$. Variants with $p_\ell=0$ or $p_\ell=1$ contribute zero to that denominator and provide no variation for this calculation; they are excluded. The formula gives greater weight to variants with more expected genotype variance than a simple unscaled sum of products would.

Assumptions and Interpretation

The calculation assumes that genotypes are coded consistently as counts of the same allele and that the frequency $p_\ell$ refers to that allele. It also assumes that the chosen variants and frequencies are appropriate for the comparison being made. For the usual interpretation of off-diagonal values as genomic relationships, the reference should represent the baseline allele frequencies of the population in which relatedness is being assessed. A single reference frequency need not be an adequate baseline for a sample that mixes ancestry groups.

The expectation makes the issue precise. Suppose two individuals are unrelated and their genotypes are independent within a population whose actual allele frequency at a variant is $q_\ell$. With reference frequency $p_\ell$, each centered genotype has mean $2(q_\ell-p_\ell)$. Independence therefore gives $$ E[(g_{i\ell}-2p_\ell)(g_{j\ell}-2p_\ell)] =4(q_\ell-p_\ell)^2. $$ This expected cross-product is positive whenever the population frequency differs from the reference. Thus unrelated members of the same population can contribute a positive relationship score simply because both are shifted in the same direction from the reference. If two groups lie on opposite sides of the reference, their expected cross-product can instead be negative.

This calculation is a justification for treating the frequency reference as part of the estimator, not as a harmless formatting choice. It does not mean every positive estimate is caused by ancestry or every negative estimate rules out relatedness. A finite set of variants produces sampling variation, and genuine relatedness, allele-frequency differences, and estimation noise can all affect the observed sum.

Worked Example: The Same Genotypes With Two References

Consider two hypothetical individuals genotyped at three biallelic variants. Their dosages are $g_X=(0,1,2)$ and $g_Y=(0,0,1)$. Compare a reference with frequencies $p=(0.25,0.50,0.75)$ to a second reference with frequencies $p=(0.50,0.50,0.50)$. These are illustrative reference panels; they are not estimates from these two individuals.

First reference. The centered vectors are $z_X=(-0.50,0,0.50)$ and $z_Y=(-0.50,-1,-0.50)$. The sum of cross-products is $ (-0.50)(-0.50)+(0)(-1)+(0.50)(-0.50)=0.25+0-0.25=0$. The denominator is $2(0.25)(0.75)+2(0.50)(0.50)+2(0.75)(0.25)=0.375+0.500+0.375=1.25$. Thus $\widehat{G}_{XY}=0/1.25=0$.

Second reference. The centered vectors are now $z_X=(-1,0,1)$ and $z_Y=(-1,-1,0)$. Their cross-products sum to $(-1)(-1)+(0)(-1)+(1)(0)=1+0+0=1$. The denominator is $3\{2(0.50)(0.50)\}=3(0.50)=1.50$, so $\widehat{G}_{XY}=1/1.50=0.667$.

The observed genotypes did not change, but the estimate did: it is 0 under the first reference and 0.667 under the second. With only three variants, these are highly illustrative, noisy estimates rather than reliable relationship classifications. The example isolates the arithmetic point: frequency choice changes both the centered products and the scale in the denominator.

Structured Samples and Apparent Relatedness

A pooled reference can be a poor baseline for every ancestry group in a structured sample. The following expectation calculation shows why, without relying on a particular small-sample realization. Suppose a variant has frequency $q_A=0.20$ in population A and $q_B=0.80$ in population B. Use a pooled reference $p=0.50$. For two unrelated people from A, the expected centered genotype for each is $2(0.20-0.50)=-0.60$. Their expected cross-product is $(-0.60)^2=0.36$, and the reference variance in the denominator is $2(0.50)(0.50)=0.50$. The expected contribution on the relationship scale is therefore $0.36/0.50=0.72$ for this one variant.

For one unrelated person from A and one from B, the expected centered genotypes are $-0.60$ and $2(0.80-0.50)=0.60$. Their expected cross-product is $(-0.60)(0.60)=-0.36$, giving a contribution of $-0.36/0.50=-0.72$. These are expectations under independent genotypes within the stated groups, not guaranteed values for particular pairs. Across variants, the summed estimates can vary, and allele frequencies need not follow this simple two-group pattern.

If the reference were instead $p=0.20$, unrelated people from A would have expected centered genotype zero, so their expected cross-product would be zero. This removes that particular frequency-shift contribution for A, but it does not make the reference suitable for every group: a B individual has expected centered genotype $2(0.80-0.20)=1.20$. A single population-specific reference can improve calibration for comparisons within that population while changing the baseline for comparisons involving another population.

Worked Example: Comparing Within- and Between-Group Expectations

Use the two population frequencies above, $q_A=0.20$ and $q_B=0.80$, with the pooled reference $p=0.50$. At this variant the denominator is $0.50$. For an unrelated A–A pair, both centered expectations are $-0.60$, so the expected numerator contribution is $(-0.60)(-0.60)=0.36$, and the scaled contribution is $0.36/0.50=0.72$. For an unrelated A–B pair, it is $(-0.60)(0.60)=-0.36$, and the scaled contribution is $-0.36/0.50=-0.72$.

Now use the A frequency as reference, $p=0.20$. The denominator becomes $2(0.20)(0.80)=0.32$. The A centered expectation is $2(0.20-0.20)=0$, while the B centered expectation is $2(0.80-0.20)=1.20$. The expected A–A cross-product is $0\times0=0$; the expected A–B cross-product is $0\times1.20=0$. In this simplified independent-genotype calculation, this reference removes the mean-shift contribution for both pair types involving an A individual. It does not establish that this reference is universally best: a comparison of B–B pairs would have expected cross-product $1.20^2=1.44$, or $1.44/0.32=4.50$ after scaling.

The large numerical contributions at one variant are not whole-genome relationship estimates. They show how a reference mismatch can affect the per-variant expectation. Averaging across many variants and accounting for their variability are essential for practical interpretation.

How to Compare Reference Choices

A useful comparison changes the frequency reference while holding the genotype data, variant set, dosage coding, and estimator fixed. Otherwise, it is not clear which change produced the difference in estimates. In structured data, candidate references might include a pooled cohort frequency, a frequency estimated within an ancestry group, or an external reference matched to the target population. Each choice answers a somewhat different calibration question.

Specify the target comparison.
State whether the aim is to assess relatedness within a defined population or across a sample containing multiple ancestry groups.

State the frequency source.
Record whether $p_\ell$ comes from a pooled sample, a relevant subgroup, or an external panel. Apply the same stated source consistently to the comparisons being reported.

Recalculate and inspect changes.
Compare pairwise estimates and centered genotypes under the candidate references. Large shifts concentrated by ancestry group indicate that frequency mismatch may be contributing to the estimates.

Report the dependence.
Describe the frequency reference and avoid presenting an estimate as an intrinsic, reference-free property of the observed pair.

A change in $p_\ell$ is different from simply changing which allele is counted. If the counted allele is swapped, the dosage becomes $g'_{i\ell}=2-g_{i\ell}$ and its frequency becomes $p'_\ell=1-p_\ell$. The centered dosage is then $(2-g_{i\ell})-2(1-p_\ell)=-(g_{i\ell}-2p_\ell)$. Both individuals' centered genotypes change sign, so their product is unchanged; the denominator $2p_\ell(1-p_\ell)$ is unchanged as well. Thus a consistent allele recoding does not change the estimate. Choosing a different frequency reference generally does.

Worked Example: Allele Recoding Is Not a New Reference

At one variant, let $g_X=0$, $g_Y=1$, and $p=0.25$. The centered genotypes are $0-2(0.25)=-0.50$ and $1-2(0.25)=0.50$. Their product is $-0.25$, and the denominator contribution is $2(0.25)(0.75)=0.375$.

If the other allele is counted instead, the dosages are $g'_X=2$, $g'_Y=1$, and the frequency is $p'=0.75$. The centered values are $2-2(0.75)=0.50$ and $1-2(0.75)=-0.50$. Their product is again $-0.25$, and the denominator remains $2(0.75)(0.25)=0.375$. The per-variant scaled contribution is $-0.25/0.375=-0.667$ under either coding. The result is invariant because the allele and frequency were recoded together.

Diagnostics, Alternatives, and Limitations

Interpret a reference-sensitive estimate as a signal to investigate, not as an automatic diagnosis. Useful checks include tabulating allele frequencies by ancestry group, comparing estimates under clearly stated references, and examining whether pairwise values track the ancestry patterns already visible in genotype covariance or principal-component analyses. If estimates shift systematically for pairs from the same group, or differ strongly between within-group and cross-group comparisons, the reference may be capturing population frequency differences as well as relatedness.

The estimator is also affected by the variants included. Linkage disequilibrium means nearby variants can carry redundant information; many correlated markers do not necessarily provide as much independent evidence as the raw marker count suggests. Rare variants have small $2p_\ell(1-p_\ell)$, making their scaled contributions potentially influential when genotypes depart from the reference expectation. Frequency estimates from small groups can themselves be noisy. These factors can make pairwise values unstable, particularly when the variant set is limited.

A common alternative is to use frequencies estimated in a target ancestry group rather than one pooled across all participants. That can better center comparisons within that group, but it changes the baseline and may not suit cross-group comparisons. An external matched reference can avoid estimating frequencies from a small target sample, but only if it is sufficiently comparable in ancestry, variant representation, and allele coding. A pooled reference is simple and consistent across the cohort, yet can leave within-group mean shifts when substantial structure is present. No choice removes the need to state the reference and define the intended comparison.

This tutorial concerns the effect of frequency choice on a conventional centered-genotype estimate. It does not provide a general correction for population structure or a rule for choosing a reference that works for every sample. In particular, a difference between estimates is not by itself proof of close relatedness or proof of a specific ancestry effect. The estimate is a summary whose meaning depends on its model, frequency baseline, and data quality.

Pitfall: treating the pooled frequency as neutral. The reasoning “the pooled frequency uses everyone, so its estimates cannot reflect ancestry” is wrong. If group frequencies differ from the pooled value, their centered genotypes have nonzero means, and unrelated pairs can have nonzero expected cross-products.
Correction: Check the relationship between frequency differences and ancestry groups. Report the reference and compare plausible alternatives when that comparison is relevant to the scientific question.
Pitfall: interpreting a small marker-set estimate as a stable relationship class. A single estimate can be noisy and reference-sensitive; it is not a direct pedigree label.
Correction: Consider the number and properties of included variants, inspect sensitivity to reference choices, and interpret the estimate in context.

Key takeaway: The allele-frequency reference determines how genotype dosages are centered and scaled. In structured samples, a mismatch between group frequencies and the chosen reference can create apparent positive or negative relationship contributions even for unrelated individuals. State the reference and interpret the estimate relative to it.

Check Your Understanding

Try each question, then check the brief solutions below.

For $g=1$ and reference frequency $p=0.30$, what is the centered genotype?
Two unrelated people come from a population with $q=0.70$, but the reference is $p=0.50$. Find the expected cross-product and its per-variant scaled contribution.
Does swapping the counted allele change the relationship estimate if dosages and frequencies are both recoded consistently? Explain.
Why might a pooled frequency create positive expected cross-products for unrelated people within one ancestry group?
Name two details that should be held fixed when comparing estimates under different frequency references.

Solutions. (1) $1-2(0.30)=0.40$. (2) Each centered expectation is $2(0.70-0.50)=0.40$, so the expected cross-product is $0.40^2=0.16$; the denominator is $2(0.50)(0.50)=0.50$, giving $0.16/0.50=0.32$. (3) No. Both centered dosages change sign, leaving their product unchanged, and $p(1-p)$ is unchanged. (4) Both individuals have expected centered dosage $2(q-p)$ in that group, so the expected cross-product is $4(q-p)^2>0$ when $q\ne p$. (5) For example, hold the genotype data and variant set fixed; also keep dosage coding and the estimator fixed.