Population Structure in Genomic Relationship Matrices

Why One Matrix Can Show Two Kinds of Similarity

A genomic relationship matrix summarizes genetic similarity among the sampled individuals. Its entries can reflect close-family relationships, but they can also reflect broad differences in allele frequencies between ancestry groups. These signals may coexist: two people can be similar because they share recent ancestors, because their ancestry is similar, or for both reasons.

This matters when interpreting the matrix’s eigenvectors. An eigenvector is a direction of variation among the sampled individuals, not a label that inherently means “ancestry” or “family.” A broad ancestry difference can produce a direction with scores spread across many individuals. Close relatives can instead contribute a more localized pattern, concentrated in a small subset. Which pattern appears first depends on its strength in the particular sample.

This definition assumes that the included variants have usable allele-frequency estimates, that the scaling denominator is nonzero, and that individuals and variants are consistently ordered in \(Z\). The precise scaling and frequency reference affect the entries and eigenvalues. The earlier material on allele-frequency choices and robust relatedness estimation is relevant here: \(G\) is an estimate built under a chosen centering and scaling convention, not a reference-free measurement of biological relatedness.

By the spectral theorem, a symmetric \(G\) has orthonormal eigenvectors \(u_r\) and real eigenvalues \(\lambda_r\). For this Gram matrix, \(\lambda_r\geq 0\). An eigenvector gives a pattern of relative scores across individuals; its eigenvalue measures the amount of matrix variation along that direction. The eigenvectors are conventionally ordered from largest to smallest eigenvalue. Their signs are arbitrary: \(u_r\) and \(-u_r\) describe the same direction.

How Ancestry and Family Covariance Enter

Consider a simplified decomposition of a standardized genotype column. Each person’s value can be written as a group-level mean plus an individual deviation from that mean. Across variants, the group means reflect systematic allele-frequency differences among ancestry groups. Deviations can include family sharing, other individual-level variation, and sampling noise.

For a precise version, suppose individuals are assigned to ancestry groups for the purpose of describing the data. Write \(Z=M+R\), where each row of \(M\) contains that individual’s group mean at each variant and \(R\) contains deviations from those means. Within every group, the deviations sum to zero at each variant. Consequently, \(M^\mathsf{T}R=0\), and

The first term is a group-mean component: individuals in groups with similar mean genotype profiles tend to have similar rows of \(M\). The second term contains variation within groups. If close relatives occur within a group, their shared genotypes can contribute strong, localized covariance to this residual component. This is a useful conceptual decomposition, not a guarantee that ancestry and family effects can always be separated exactly: family members may span groups, groups may be uncertain, and the chosen allele-frequency reference affects both terms.

The key algebra is that the cross-products vanish under the stated group-centering condition. Expanding gives \(ZZ^\mathsf{T}=MM^\mathsf{T}+MR^\mathsf{T}+RM^\mathsf{T}+RR^\mathsf{T}\). Since \(M^\mathsf{T}R=0\), both cross terms are zero, leaving the two components above. Without that condition, the cross terms need not vanish, so this exact split should not be assumed for arbitrary definitions of “ancestry mean” and “residual.”

Reading Eigenvectors as Patterns Across Individuals

An eigenvector identifies a pattern that explains a large part of the matrix’s variation. If many individuals in one ancestry group have scores of one sign and many in another group have scores of the opposite sign, the pattern may reflect broad ancestry structure. If only a few individuals have substantial scores, especially when they are relatives, the pattern may be family-localized. The interpretation depends on checking the people’s positions and relationships, not on the eigenvector’s rank alone.

For an individual-level coordinate plot, one common convention uses \(\sqrt{\lambda_r}u_r\) as the coordinate on eigenvector \(r\). Other software may report a rescaled version. The relative pattern is the important point here; comparisons of numerical coordinates require knowing the software’s scaling convention. A large eigenvalue says that a direction is prominent in this sample, not that it has a particular biological cause.

Practical Interpretation, Diagnostics, and Alternatives

Interpretation should be tied to the sample and the construction of \(G\). Linkage disequilibrium pruning, genotype scaling, allele-frequency estimates, and sample composition can change which directions are prominent. A few groups with strong frequency differences can produce a broad leading direction; a set of close relatives can contribute a concentrated one. An eigenvector may also combine these patterns rather than represent either cleanly.

Useful checks include comparing eigenvector scores with known or independently assessed ancestry information, checking whether large absolute scores are concentrated among a few individuals, and examining whether those individuals are close relatives. Also inspect whether the broad pattern is supported across many variants rather than driven by a small subset. These checks help distinguish plausible structure from artifacts, but they do not convert an eigenvector into proof of ancestry or kinship.

The genotype-based PCA approach and the GRM eigenvector approach are closely connected: when \(G=ZZ^\mathsf{T}/m\), they describe the same sample-side directions, subject to coordinate scaling. A direct pairwise relatedness analysis instead focuses on particular entries or pairs and can be more targeted for identifying close relationships. Pedigree kinship uses recorded family relationships and a model of inheritance rather than estimating similarity from the observed genotype matrix. These approaches answer related, but not identical, questions.

The main limitation is that eigenvectors summarize variation, not its cause. A strong ancestry component and a strong family component can overlap, while a weak component may be hard to interpret in a small or unbalanced sample. Estimates can also be affected by the frequency reference and the variants included. Therefore, report what the pattern looks like—broad across groups, or concentrated among a few individuals—and state the evidence used for its interpretation.

Common Mistake