Why Genotype Covariance Reveals Population Structure

Why Shared Allele-Frequency Differences Create Covariance

A genotype matrix has individuals in rows and genetic variants in columns. After centering each variant, some individuals may show similar patterns of positive and negative dosage deviations across many variants. This similarity is not necessarily evidence that they are close relatives. It can arise because their ancestry proportions are similar and ancestry groups differ in allele frequencies. The resulting covariance pattern is useful because it compresses many small frequency differences into a shared signal across individuals.

The argument begins with a population mixture model. At locus \(j\), let \(p_{jk}\) be the frequency of the counted allele in ancestry group \(k\). Individual \(i\) has ancestry proportion \(q_{ik}\) from group \(k\), with \(q_{ik}\geq 0\) and \(\sum_{k=1}^K q_{ik}=1\). Let \(G_{ij}\in\{0,1,2\}\) be the diploid allele dosage. Under random union of gametes and the mixture model, the expected dosage is twice the individual's ancestry-weighted allele frequency.

Here the reference proportions sum to one. Centering by \(2\bar p_j\) removes the reference dosage at each locus. If an individual's ancestry proportions differ from the reference, their expected centered dosage need not be zero. The important point is that the differences can recur in a coordinated way across loci: the same ancestry contrast affects every locus where the groups have different allele frequencies.

Deriving the Cross-Individual Covariance

For a transparent covariance definition, consider the expected centered profiles across \(m\) loci, and use divisor \(m\). For individuals \(i\) and \(\ell\), define \(C_{i\ell}=m^{-1}\sum_{j=1}^m d_{ij}d_{\ell j}\). This is an average cross-product across loci, not the sampling covariance of two random genotypes at one locus. Positive values indicate that the expected centered profiles tend to deviate in the same direction across loci; negative values indicate opposing deviations.

For two ancestry groups, write \(q_i\) for individual \(i\)'s proportion from group 1, so their proportion from group 2 is \(1-q_i\). Let \(\delta_j=p_{j1}-p_{j2}\) and let \(\bar q\) be the reference proportion from group 1. The reference frequency is \(\bar p_j=\bar q p_{j1}+(1-\bar q)p_{j2}\). Subtracting this reference from the individual's weighted frequency gives:

Multiplying by two gives \(d_{ij}=2(q_i-\bar q)\delta_j\). Substituting this expression into the covariance definition yields the key result:

The average squared allele-frequency contrast is nonnegative. Thus, in this two-group model, individuals whose ancestry proportions lie on the same side of the reference tend to have positive expected covariance, while individuals on opposite sides tend to have negative covariance. The magnitude increases with both their distances from the reference and the amount of allele-frequency differentiation across the analyzed loci. If the groups have identical frequencies at all included loci, every \(\delta_j=0\), and this ancestry signal is zero.

With \(K\) groups, the same logic applies to the vector of ancestry deviations. The matrix with entries \(C_{i\ell}\) is a Gram matrix of expected centered profiles: it is \(DD^\mathsf{T}/m\), where \(D\) contains the \(d_{ij}\). It is therefore positive semidefinite. Because ancestry proportions sum to one, their deviations have at most \(K-1\) independent dimensions. Consequently, the ancestry-driven profile matrix has rank at most \(K-1\), and may have lower rank if the included loci do not distinguish all ancestry contrasts. This explains why a small number of broad covariance patterns can summarize variation spread across many loci. It does not imply that every observed covariance pattern is ancestry, or that a particular component has a unique biological interpretation.

From the Model to Observed Genotype Data

Observed genotypes include allele-sampling variation around their expected dosage profiles. To see the distinction, suppose genotypes are conditionally independent between unrelated individuals given their ancestry proportions and the locus-specific frequencies. For two distinct individuals, conditional independence implies that the expected product of their centered dosages at a locus equals the product of their expected centered dosages: \(E[(G_{ij}-2\bar p_j)(G_{\ell j}-2\bar p_j)\mid q_i,q_\ell]=d_{ij}d_{\ell j}\). Averaging over loci therefore recovers the ancestry-profile cross-product in expectation, provided the loci are treated appropriately and the reference frequencies are suitable.

For an individual's own squared centered dosage, however, genotype sampling variance contributes in addition to the squared expected deviation. The diagonal of an observed covariance matrix therefore contains both ancestry-related variation and within-individual genotype variation. Related individuals can also have correlated genotype residuals, violating the conditional independence assumption for distinct individuals. These distinctions matter: covariance across individuals is a useful signal, not a noise-free measurement of ancestry.

Assumptions, Diagnostics, and Alternatives

The derivation describes an ancestry-frequency signal under a specified mixture model. Its interpretation is strongest when allele dosages are correctly coded, ancestry-specific frequency differences are real, and technical artifacts do not create systematic differences among individuals. The conditional-independence argument for distinct individuals additionally assumes no residual genotype dependence from close relatedness. In practice, these conditions are approximations, not guarantees.

• Check allele and sample quality. Genotype coding errors, strand mismatches, sample swaps, and differential missingness can produce coherent-looking but artificial patterns.
• Check technical structure. If sequencing or genotyping batch is associated with sample ancestry, batch can contribute covariance. Earlier material on batch effects emphasizes that technical and biological patterns can be confounded rather than separable from the data alone.
• Consider linkage disequilibrium. Nearby variants may carry redundant information. Treating many correlated loci as independent can make some genomic regions dominate the covariance summary.
• Inspect scaling and frequency choices. Centering removes a locus's reference mean; scaling changes the relative influence of loci. Rare variants can receive large weight under some scaling choices, while common variants may dominate unscaled dosage covariance.
• Account for relatedness and sampling. Close relatives share genotypes beyond what broad ancestry proportions predict. A small or unrepresentative sample may also make the reference frequency a poor description of the intended population.

Covariance is not the only way to describe structure. Direct allele-frequency comparisons between groups are easier to interpret at named loci but require group labels and do not automatically summarize many dimensions. Pairwise genetic distances offer an individual-level dissimilarity measure, but do not by themselves identify which allele-frequency contrasts drive that dissimilarity. Covariance aggregates signed deviations across loci, preserving whether profiles tend to move together or oppositely. Its compactness is useful, but its interpretation depends on the chosen loci, centering, scaling, and sample.

A common mistake is to say, “negative covariance proves these people come from different populations.” The calculation only says their centered dosage profiles tend to point in opposite directions relative to the chosen reference. A continuous ancestry gradient, imperfect reference frequencies, technical artifacts, or sampling variation can all generate or alter that pattern. Covariance is evidence about shared variation in the data, not a direct label for an individual's ancestry or proof of a causal account.