Testing Allele-Frequency Differences Between Two Populations

Comparing Allele Frequencies

An allele-frequency comparison asks whether a specified allele has the same population frequency in two groups. For example, at a biallelic locus, one might compare the frequency of allele A in population 1 with its frequency in population 2. The primary effect size is the difference in those frequencies. A test addresses a separate question: how incompatible are the observed counts with equality, under a specified sampling model?

The distinction matters because a small frequency difference can be estimated precisely in a large sample and yield strong statistical evidence against equality. Conversely, a large observed difference in a small sample can be imprecise. Neither a p-value nor a test decision gives the size or practical importance of the frequency difference on its own.

For diploid samples with complete genotypes, \(m_j=2n_j\), where \(n_j\) is the number of individuals. The simple binomial formulas below treat the \(m_j\) allele copies as independent Bernoulli observations with success probability \(p_j\). This is appropriate, for example, for independently sampled chromosomes, or for allele copies from unrelated individuals under random pairing and Hardy–Weinberg proportions. If the two copies within an individual are correlated because of inbreeding or another departure from this model, the binomial standard error and tests need an appropriate adjustment; counting two copies per person does not by itself make them independent. The sampling design must also support inference to the populations being compared.

Estimating the Difference and Its Uncertainty

Under independent binomial sampling in the two populations, \(\hat{p}_1\) and \(\hat{p}_2\) are independent and have variances \(p_1(1-p_1)/m_1\) and \(p_2(1-p_2)/m_2\), respectively. Thus the variance of their difference is the sum of those variances. Replacing the unknown population frequencies by their sample estimates gives the usual estimated standard error:

When sample sizes are adequate and neither frequency is too close to 0 or 1, a Wald confidence interval is \(\hat{\Delta}\pm z^*SE(\hat{\Delta})\), where \(z^*\) is the standard normal critical value (1.96 for a two-sided 95% interval). This is an approximate interval, and can perform poorly with sparse counts or frequencies near a boundary. A confidence interval describes values of the population difference compatible with the data and the model; it does not assign a probability to the fixed parameter after the interval has been calculated.

Testing Equality With a Pooled Score Test

For a two-sided comparison, the hypotheses are \(H_0:p_1=p_2\) against \(H_a:p_1\ne p_2\). Under \(H_0\), both groups share one frequency \(p\). The maximum-likelihood estimate of that common frequency is the pooled frequency, which combines the allele counts and the total number of copies. The null standard error uses this pooled estimate, not the two separate sample estimates.

For a two-sided test, the approximate p-value is \(2P(Z_{\mathrm{std}}\ge |z_{\mathrm{obs}}|)\), where \(Z_{\mathrm{std}}\sim N(0,1)\). At significance level \(\alpha\), reject \(H_0\) if the p-value is at most \(\alpha\). This approximation relies on the binomial sampling model and sufficiently large expected allele and non-allele counts in both groups. When counts are sparse, an exact method may be preferable.

The score standard error is evaluated under the null because the test asks how variable the difference would be if the frequencies were equal. It is therefore not the same quantity as the unpooled standard error used for the Wald interval. A test and an interval can use different approximations; their numerical results need not match exactly.

Why the Score Formula Works

The pooled score test follows from the sampling variance of a difference under the null. If the common frequency is \(p\), then each independent sample frequency has variance \(p(1-p)/m_j\). Independence of the samples means the variance of their difference is the sum:

The common \(p\) is unknown, so the test estimates it under \(H_0\) by \(\hat{p}_{\mathrm{pool}}\). Dividing the observed difference by the square root of this estimated null variance standardizes the difference. For sufficiently large samples, the standardized statistic is approximately standard normal under \(H_0\). This justification also identifies the central assumption: the variance formula is not valid if allele observations have dependence that has not been accounted for.

Exact Conditional Testing for Small Counts

For small samples, the normal approximation can be unreliable. Under independent binomial sampling and \(H_0\), conditioning on the total number of A copies removes the unknown common frequency. Given \(x_1+x_2\), the number of A copies allocated to group 1 has a hypergeometric distribution: each possible allocation is evaluated according to the fixed group sizes and total allele count. Summing the probabilities of allocations at least as incompatible with equality as the observed allocation gives a conditional exact p-value. For a two-sided Fisher exact test, software commonly sums probabilities of tables whose probability is no greater than that of the observed table; two-sided exact conventions can differ, so report the convention used.

Interpreting Results and Avoiding Pitfalls

Report the estimated frequencies, the signed difference, and an uncertainty summary alongside the test result. A p-value is the probability, assuming \(H_0\) and the model, of a test statistic at least as extreme as the observed statistic; it is not the probability that \(H_0\) is true. Rejecting \(H_0\) does not measure biological importance, and failing to reject it does not demonstrate that the frequencies are equal. The direction is carried by \(\hat{\Delta}\), while the two-sided p-value measures evidence against equality without encoding that direction by itself.

An exact conditional test is not automatically valid for every diploid genotype table. The hypergeometric calculation above treats allele copies as independent observations. If both alleles from each person are counted and within-person dependence is present, conditioning on the total allele count does not repair that dependence. Use a variance or testing procedure that respects the individual-level sampling structure—for example, an appropriate inbreeding-adjusted method when its assumptions apply. Likewise, a population sample affected by nonrepresentative sampling can give a precise answer for the sampled individuals but a biased estimate of the target population frequency.

• Pitfall: using separate standard errors for a null test. The reasoning “the two observed frequencies have separate variances, so use the unpooled standard error for the equality test” overlooks that the score test evaluates variation under \(H_0\). The correction is to use the pooled frequency in the null standard error; use the unpooled formula for the usual Wald interval.
• Pitfall: treating every allele copy as independent. Two allele copies from one diploid individual can be correlated. The correction is to state the sampling model and use an adjustment or individual-level method if that independence assumption fails.
• Pitfall: equating a small p-value with a large effect. A large sample can make a modest difference statistically detectable. Always report \(\hat{\Delta}\) in frequency units (or percentage points), with uncertainty, separately from the p-value.