Rare Alleles And Frequency-Sensitive Estimators

Why Allele Frequency Changes a Relatedness Estimate

Pairwise relatedness estimators do not treat every genotype pattern identically. In a frequency-standardized genomic relationship matrix, each marker's contribution is centered on its expected genotype count and scaled by the genotype variance implied by its allele frequency. This scaling gives less common alleles the potential to contribute more strongly than common alleles. That can make rare variants informative, but it can also make estimates sensitive to a small number of carriers and to error in the estimated frequency.

The issue is not simply that a rare allele is unusual. The issue is how an estimator converts a genotype deviation into a standardized contribution. Understanding that conversion helps explain why a pairwise estimate can move substantially when a rare variant is included, removed, or assigned a different frequency.

Definition: At a biallelic marker, let $p$ be the frequency of the allele chosen for counting, with $0<p\leq 0.5$, and let $x_i\in\{0,1,2\}$ be the number of counted alleles for individual $i$. Under a model in which the two alleles are independent draws with frequency $p$, the expected genotype count is $2p$ and its variance is $2p(1-p)$. The frequency-standardized contribution of this marker to the pairwise genomic relationship estimate for individuals $i$ and $j$ is $$ K_{ij,\ell}=\frac{(x_{i\ell}-2p_\ell)(x_{j\ell}-2p_\ell)}{2p_\ell(1-p_\ell)}. $$ For $m$ included markers with complete genotypes, the estimate is $K_{ij}=m^{-1}\sum_{\ell=1}^{m}K_{ij,\ell}$. It is a dimensionless estimate on the genomic-relationship scale; it is not automatically identical to every kinship-coefficient convention.

The formula makes two operations visible. Subtracting $2p$ centers each genotype around its frequency-specific expectation. Dividing by $2p(1-p)$ scales the product by the expected genotype variance. As $p$ approaches zero, this variance becomes small. Therefore, a given genotype pattern can yield a larger standardized contribution at a rare marker than at a common marker.

This model-based scaling relies on the frequency being appropriate for the individuals being compared and on the variance model being a reasonable reference. The examples below assume a single comparable group and use the same allele coding for both individuals. They illustrate the estimator's frequency sensitivity, not a claim that a large marker contribution is automatically reliable evidence of close biological relationship.

How Rare-Allele Genotypes Affect a Marker Contribution

At a rare marker, most individuals have genotype count zero. If both members of a pair have zero copies, their centered values are both $-2p$, so the contribution is positive but small when $p$ is small. If both carry one copy, each centered value is close to one, and their product is divided by a small variance. That can produce a large positive contribution. If only one carries the allele, the centered values have opposite signs, and the contribution can instead be negative.

Worked Example: Comparing a Rare and a Common Marker

Suppose two individuals each have one copy of the counted allele at a marker with frequency $p=0.01$. Their genotype counts are $x_i=x_j=1$. The centered values are $1-2(0.01)=0.98$, and the genotype variance used for scaling is $2(0.01)(0.99)=0.0198$. Thus $$ K_{ij,\ell}=\frac{(0.98)(0.98)}{0.0198} =\frac{0.9604}{0.0198}\approx48.51. $$

At a common marker with $p=0.30$, suppose both individuals again have one copy. The centered values are $1-0.60=0.40$, and the scaling variance is $2(0.30)(0.70)=0.42$. The contribution is $$ K_{ij,\ell}=\frac{(0.40)(0.40)}{0.42} =\frac{0.16}{0.42}\approx0.3810. $$

The rare-marker contribution is much larger for this particular pair of genotype patterns. It is not a stand-alone relatedness estimate: the matrix estimate averages contributions over included markers. A large contribution can nevertheless have substantial influence, especially if only a modest number of markers are included or if the variant is carried by very few sampled individuals.

A large positive contribution is not guaranteed at a rare marker. For example, with $p=0.01$, if $x_i=1$ and $x_j=0$, the centered values are $0.98$ and $-0.02$. The contribution is $0.98(-0.02)/0.0198\approx-0.9899$. Thus rare-allele carrier patterns can generate contributions with either sign; their potential magnitude depends on the observed genotypes as well as the frequency.

Frequency Estimation Error and Instability

In practice, the population frequency is not known. An analysis typically substitutes an estimated frequency $\hat p$ for $p$, using it both to center the genotype counts and to scale the contribution. At a rare marker, a small change in $\hat p$ can be large relative to the frequency itself. It also changes the small denominator $2\hat p(1-\hat p)$, so the resulting pairwise contribution may change markedly.

The following calculation describes sampling uncertainty under a simple model: $N$ independently sampled chromosomes, each with counted-allele probability $p$. Then the estimated frequency is the allele count divided by $N$, with variance $p(1-p)/N$. Real genotype data can depart from this simple sampling model, but it illustrates why a low frequency estimated from very few copies is imprecise.

Worked Example: A Small Frequency Change at a Rare Marker

Suppose the counted-allele frequency is estimated from $N=100$ chromosomes. Under the independent sampling model, the standard error of $\hat p$ is $\sqrt{p(1-p)/N}$. If $p=0.01$, this is $$ SE(\hat p)=\sqrt{\frac{(0.01)(0.99)}{100}} =\sqrt{0.000099}\approx0.009950. $$ This standard error is nearly as large as the frequency itself. For comparison, at $p=0.25$, the corresponding standard error is $$ SE(\hat p)=\sqrt{\frac{(0.25)(0.75)}{100}} =\sqrt{0.001875}\approx0.04330, $$ which is smaller relative to $0.25$.

To see the consequence for a pairwise contribution, suppose both individuals carry one copy. If one counted copy is observed among 100 chromosomes, then $\hat p=0.01$, and the contribution is $$ \frac{(1-2(0.01))^2}{2(0.01)(0.99)} =\frac{0.9604}{0.0198}\approx48.51. $$ If instead two copies are observed, then $\hat p=0.02$, giving $$ \frac{(1-2(0.02))^2}{2(0.02)(0.98)} =\frac{0.9216}{0.0392}\approx23.51. $$ The observed frequency has doubled, but the contribution has fallen by about half. The arithmetic shows how frequency estimation can affect a pairwise term; it does not imply that the two frequency estimates are interchangeable or that either estimate is precise.

This sensitivity is one reason to inspect allele counts, not only reported frequencies. A frequency such as $0.01$ can correspond to very few observed copies in the reference sample. Estimates based on few copies are particularly vulnerable to sampling variation, genotyping error, and differences between the reference sample and the individuals being compared. Since the same estimated frequency enters the centering and scaling, uncertainty can affect both parts of the contribution.

Common-Variant Filters and Frequency Caps

Two practical approaches can reduce the influence of low-frequency markers, but they make different choices. A common-variant filter excludes markers below a specified frequency threshold. A frequency cap retains a marker but limits how large its inverse-variance scaling can become. The details of a cap must be stated: here, the cap limits the denominator used for scaling while retaining the observed frequency for centering.

Approach	What happens to a low-frequency marker?	Main consequence
Common-variant filter	Markers with $\hat p<p_{\min}$ are omitted.	Changes the set of markers being averaged; omitted markers contribute nothing.
Frequency cap	The marker remains, but the scaling denominator is prevented from falling below a chosen value.	Retains some genotype information while limiting the marker's maximum standardized influence.

For example, with a minimum frequency $p_{\min}=0.05$, a filter excludes a marker estimated at $\hat p=0.01$. Under the cap defined here, the denominator $2\hat p(1-\hat p)$ is replaced by the larger of itself and $2p_{\min}(1-p_{\min})$. The genotype counts are still centered using $\hat p$. A cap is not the same as replacing $\hat p$ with $p_{\min}$ everywhere; other implementations may define a frequency floor differently and should be described explicitly.

Worked Example: Filtering Versus Capping a Rare Marker

Suppose $\hat p=0.01$, both individuals have one counted allele, and $p_{\min}=0.05$. Without a cap, the contribution is $0.9604/0.0198\approx48.51$. For the cap defined above, the minimum allowed denominator is $$ 2(0.05)(0.95)=0.095. $$ Since $0.095>0.0198$, the capped contribution is $$ K_{ij,\ell}^{\mathrm{cap}}=\frac{(1-2(0.01))^2}{0.095} =\frac{0.9604}{0.095}\approx10.11. $$

The cap reduces this marker's contribution while retaining it in the analysis. A common-variant filter with the same threshold instead excludes the marker because $0.01<0.05$; it would not assign this marker a contribution at all. If filtering is used, the estimate is averaged over the remaining markers. If a cap is used, the marker remains among the included markers, and the capped contribution enters the average.

Neither choice is universally preferable. Filtering can make an estimate less dependent on very rare variants but discards their information. Capping retains some information but changes the scaling rule. The threshold or cap should be chosen and reported as an analysis decision, not presented as a fact about the genetic relationship.

Interpretation and Practical Checks

Frequency-sensitive contributions matter because a pairwise estimate is an aggregate of marker-level terms. A small number of large terms can influence that aggregate, and markers in linkage disequilibrium are not independent pieces of evidence. Earlier material on effective marker counts addresses why the raw number of markers need not equal the amount of independent information. Here, the additional concern is that rare-allele terms may be especially sensitive to the observed carriers and to the frequency supplied to the estimator.

Before interpreting an estimate, check which allele-frequency reference was used, how low-frequency markers were handled, and whether the comparison uses an appropriate frequency model. Compare results under clearly specified filtering or capping choices when rare variants could drive the result. Such comparisons are sensitivity checks: stability across choices supports robustness to that particular choice, but does not prove that all assumptions hold.

Inspect the frequency and allele count.
Identify markers whose estimated frequencies are based on very few counted alleles; a low frequency alone does not show how precisely it is known.

Specify the estimator's frequency rule.
State whether frequencies are estimated from the analyzed sample or another reference, and define any filter or cap precisely.

Check sensitivity and report the choice.
Compare estimates under defensible frequency-handling choices when relevant, and report the marker set and rule used.

Common Mistake

A common mistake is to reason, “A rare allele is observed in both people, so it must be strong evidence that they are closely related.” The correction is that the estimator assigns a potentially large standardized marker contribution to that genotype pattern because the scaling variance is small. One contribution is not a relationship classification, and its size can be sensitive to frequency estimation. The overall estimate averages marker contributions and should be interpreted with its frequency choices and assumptions in view.

A related mistake is to treat a frequency cap as equivalent to filtering. They are not equivalent: filtering removes the marker and changes the included set, whereas a cap retains the marker with modified scaling. The analysis description should say which operation was performed, what threshold was used, and how frequencies entered the calculation.

Key takeaway: Frequency standardization can make rare-allele carrier patterns influential because their expected genotype variance is small. Rare-marker contributions can be useful, but their sensitivity to observed carriers and estimated frequencies makes frequency filters, caps, and transparent sensitivity checks important.

Check Your Understanding

Answer each question using the frequency-standardized contribution defined above.

Why can two individuals who both carry one copy at a rare marker have a large positive marker contribution?
At a marker with $p=0.01$, what is the sign of the contribution if one individual has genotype count 1 and the other has count 0? Explain using the centered genotype values.
Under independent chromosome sampling, why is a frequency estimate based on very few copies relatively uncertain?
With a cap that modifies only the scaling denominator, what happens to a marker below the chosen minimum frequency, and how does that differ from filtering it out?
Why does a large single-marker contribution not, by itself, establish that a pair is closely related?

Approach	What happens to a low-frequency marker?	Main consequence
Common-variant filter	Markers with \(\hat p<p_{\min}\) are omitted.	Changes the set of markers being averaged; omitted markers contribute nothing.
Frequency cap	The marker remains, but the scaling denominator is prevented from falling below a chosen value.	Retains some genotype information while limiting the marker's maximum standardized influence.