Exact Tests For Two-Group Count Comparisons

Why Condition on the Combined Count?

For one gene measured in two independent groups, the question is often whether its normalized expression differs between groups. A negative binomial model allows the variance of a count to exceed its mean, a feature common in RNA-seq data. In a restricted two-group setting, we can test the null hypothesis of equal group means by conditioning on the gene's combined count. This removes the unknown common mean from the null distribution and produces a discrete exact test.

The restriction matters. The simple conditional test below assumes equal effective exposure for every sample, independent observations, and a common dispersion treated as known. It does not automatically accommodate unequal library-size offsets, covariates, or paired observations. A negative binomial GLM can represent a broader design and tests a model coefficient instead.

Definition: Use the negative binomial parameterization $Y\sim NB(r,\mu)$, with mean $\mu$, size $r$, and variance $\mu+\mu^2/r$. Write the dispersion as $\phi=1/r$, so $\operatorname{Var}(Y)=\mu+\phi\mu^2$. For the conditional exact test, assume each of $m_1$ samples in group 1 and $m_2$ samples in group 2 has the same effective exposure, counts are independent, and all samples share the same dispersion $\phi$. Under $H_0$, both groups have the same per-sample mean $\mu$.

Let $S_1$ and $S_2$ be the sums of counts in the two groups, and let $T=S_1+S_2$. Under the stated model, $S_1\sim NB(m_1r,m_1\mu)$ and $S_2\sim NB(m_2r,m_2\mu)$, using the same parameterization. The group sums are independent. The null hypothesis is $H_0:\mu_1=\mu_2$; a directional alternative, for example, is $H_a:\mu_1>\mu_2$.

Theorem: Under $H_0$, conditional on $T=t$, the group 1 sum has a beta-binomial distribution: $$ P(S_1=x\mid T=t,H_0) = \binom{t}{x}\frac{B(x+a,t-x+b)}{B(a,b)}, \qquad x=0,\ldots,t, \qquad a=\frac{m_1}{\phi},\quad b=\frac{m_2}{\phi}. $$ Here $B(u,v)=\Gamma(u)\Gamma(v)/\Gamma(u+v)$ is the beta function. The conditional distribution depends on group sizes and dispersion, but not on the unknown common mean $\mu$.

Why the Conditional Distribution Is Beta-Binomial

A negative binomial variable with size $r$ and mean $\mu$ has probability mass function $$ P(Y=y)=\frac{\Gamma(y+r)}{\Gamma(r)y!} \left(\frac{r}{r+\mu}\right)^r \left(\frac{\mu}{r+\mu}\right)^y. $$ For independent samples with the same mean and dispersion, group totals also have negative binomial distributions: their sizes are $m_1r$ and $m_2r$, and they share the same factor $\mu/(r+\mu)$ raised to the observed count. In the joint probability of $S_1=x$ and $S_2=t-x$, the terms involving that factor combine to $\{\mu/(r+\mu)\}^t$. They are constant across possible allocations $x=0,\ldots,t$. Dividing by the probability of $T=t$ cancels this common-mean term.

The remaining terms are proportional to $\Gamma(x+a)\Gamma(t-x+b)/(x!(t-x)!)$, where $a=m_1r$ and $b=m_2r$. Normalizing these weights over all $x$ gives the beta-binomial formula above. This conditioning argument is the reason the null distribution does not require an estimate of $\mu$.

As $\phi$ approaches zero, $a$ and $b$ grow while their ratio remains $m_1:m_2$. The beta-binomial distribution then approaches a binomial distribution with $t$ trials and success probability $m_1/(m_1+m_2)$. With positive dispersion, it is more spread out: under the beta-binomial representation, the allocation probability itself varies, reflecting extra-Poisson variation.

Worked Example: A Two-Sided Exact Test

Suppose two groups each contain two hypothetical samples, all with equal effective exposure. Let the common dispersion be $\phi=1$. The observed group totals are $S_1=4$ and $S_2=0$, so $t=4$. Under the null, $a=b=2$. The conditional probability of $S_1=x$ is $$ P(S_1=x\mid T=4,H_0)=\binom{4}{x}\frac{B(x+2,6-x)}{B(2,2)}. $$

For $x=0$, this is $B(2,6)/B(2,2)=(1/42)/(1/6)=1/7$. For $x=1$, it is $4B(3,5)/B(2,2)=4(1/105)/(1/6)=8/35$. By symmetry, the probabilities for $x=3$ and $x=4$ are $8/35$ and $1/7$. For $x=2$, the probability is $6B(4,4)/B(2,2)=6(1/140)/(1/6)=9/35$. These sum to $1$.

Using the probability-ordering definition of a two-sided exact p-value, sum the probabilities of outcomes no more likely under $H_0$ than the observed outcome. The observed outcome has probability $1/7$, so only $x=0$ and $x=4$ qualify: $$ p_{\mathrm{two-sided}}=P(S_1=0\mid T=4)+P(S_1=4\mid T=4)=\frac{1}{7}+\frac{1}{7}=\frac{2}{7}\approx 0.286. $$ The one-sided p-value for the alternative that group 1 has the larger mean is $P(S_1\ge4\mid T=4)=1/7\approx0.143$. These p-values do not provide evidence against equal means at the 0.05 level.

Calculating and Interpreting Exact P-Values

For a prespecified alternative $\mu_1>\mu_2$, the upper-tail p-value is the conditional probability of observing a group 1 total at least as large as the one seen: $$ p_{\mathrm{upper}}=\sum_{x=S_{1,\mathrm{obs}}}^{t}P(S_1=x\mid T=t,H_0). $$ For $\mu_1<\mu_2$, use the lower tail instead. A two-sided exact p-value needs a stated convention because the null distribution is discrete and not necessarily symmetric. One common convention, used in the first example, sums probabilities of all outcomes whose null probability is no greater than that of the observed outcome. Other conventions exist; software output should be interpreted using its stated definition.

An exact p-value is calculated from the specified conditional null distribution, rather than from a continuous approximation. It is therefore valid at or below a chosen significance level under the model assumptions, but its discreteness can make it conservative: attainable p-values may skip over $\alpha$. A mid-p calculation assigns half the probability of the observed outcome to a tail. It can reduce conservatism, but it is not an exact test in the same guaranteed type-I-error sense.

Worked Example: Unequal Numbers of Samples

Consider hypothetical equal-exposure groups with $m_1=1$, $m_2=2$, and common dispersion $\phi=1$. Suppose the observed totals are $S_1=2$ and $S_2=1$, so $t=3$. Then $a=1$, $b=2$. The conditional probabilities for $x=0,1,2,3$ are: $$ \begin{aligned} P(S_1=0\mid T=3)&=\frac{B(1,5)}{B(1,2)}=\frac{1/5}{1/2}=0.4,\\ P(S_1=1\mid T=3)&=3\frac{B(2,4)}{B(1,2)}=3\frac{1/20}{1/2}=0.3,\\ P(S_1=2\mid T=3)&=3\frac{B(3,3)}{B(1,2)}=3\frac{1/30}{1/2}=0.2,\\ P(S_1=3\mid T=3)&=\frac{B(4,2)}{B(1,2)}=\frac{1/20}{1/2}=0.1. \end{aligned} $$

The one-sided upper-tail p-value for the observed $S_1=2$ is $0.2+0.1=0.3$. For the probability-ordered two-sided convention, outcomes $x=2$ and $x=3$ have probabilities no greater than the observed probability $0.2$, giving $p=0.2+0.1=0.3$. The unequal group sizes are accounted for by $a=1$ and $b=2$; a symmetric null distribution would be inappropriate.

Assumptions, Diagnostics, and Failure Modes

The word “exact” refers to calibration under the specified conditional null model. It does not mean that the test is assumption-free. Before using this test, check that the comparison is genuinely an independent two-group comparison, with no unmodeled pairing or covariates, and that the samples have equal effective exposure or satisfy the particular conditional model being used. Also check that a common dispersion is defensible for the gene and that the dispersion value can be treated as fixed for the test.

In practice, dispersion is often estimated from data and may be shared or stabilized across genes. Plugging an estimated value into the beta-binomial probabilities is useful, but the resulting p-value is not strictly exact with respect to the uncertainty in that estimate. Strong outliers, substantial sample-to-sample heterogeneity beyond the assumed negative binomial model, or different dispersions between groups can also undermine calibration. These concerns call for model checks and a method suited to the design; they cannot be repaired simply by labeling the calculation exact.

Worked Example: Why Unequal Exposures Change the Target

Suppose hypothetical group A samples have counts $10,20$ and effective exposures $1,2$, while group B samples have counts $30,40$ and exposures $3,4$, in matching library-size units. The raw totals are $30$ and $70$, which might appear to suggest higher expression in group B. But the group exposure totals are $3$ and $7$, so the exposure-adjusted rates are $$ \widehat{\lambda}_A=\frac{10+20}{1+2}=10,\qquad \widehat{\lambda}_B=\frac{30+40}{3+4}=10. $$ The estimated log rate ratio is $\log(\widehat{\lambda}_B/\widehat{\lambda}_A)=\log(10/10)=0$, not a positive effect.

A negative binomial GLM with log exposure as an offset targets this rate comparison: its group coefficient is the log rate ratio, which is zero for these data. The simple conditional beta-binomial calculation in this tutorial assumes equal effective exposure and cannot be applied as though the two raw totals had the same exposure. This example shows why a raw-total comparison is not a substitute for accounting for library size.

Conditional Test and GLM Coefficient Test

A two-group negative binomial GLM with a log link expresses the mean as a function of a group indicator and, when needed, an offset for exposure. With group 1 as the reference, the group coefficient $\beta_1$ is the log ratio of the groups' mean rates when the exposure offset is included. Its coefficient null is $H_0:\beta_1=0$, equivalent to a rate ratio of one. Wald or likelihood-ratio procedures assess that coefficient using the fitted GLM and its assumptions.

The conditional exact test asks a more restricted question: under the common-mean null and its equal-exposure assumptions, is the observed allocation of the fixed combined count unusually imbalanced between the groups? Conditioning removes the common baseline mean. The GLM coefficient test instead targets a model coefficient and can include offsets and covariates; its reference distribution is generally based on an approximation or likelihood theory. Neither target is universally preferable: the design, assumptions, and intended effect determine which is appropriate.

Feature	Conditional exact test	Negative binomial GLM coefficient test
Null target	Equal group means under the conditional model	Specified coefficient is zero, such as a log rate ratio
Baseline mean	Removed by conditioning on the combined count	Estimated as part of the fitted model
Design scope	Simple independent two-group setting	Can represent offsets and covariates in the design
Calibration	Finite-sample conditional calculation when assumptions and dispersion are fixed	Usually relies on a model-based asymptotic test
Typical limitation	Discrete p-values; restricted exposure and design assumptions	Approximation and model-fit concerns, especially with limited data

Common Pitfall

Wrong reasoning: “The test is exact, so it remains exact if I use it on any two groups, even when library sizes differ or dispersion has been estimated.” Correction: exactness is conditional on the model used to derive the null distribution. Unequal exposures change the allocation model, and estimated dispersion adds uncertainty not included in the fixed-dispersion calculation. Confirm the assumptions or use a model that represents the actual design.

A second error is to interpret a p-value as the probability that the null hypothesis is true. A p-value is instead the probability, assuming the null model and conditioning information are correct, of obtaining an outcome at least as extreme according to the chosen tail convention. It is not a probability of the hypothesis itself, nor does it measure the size of a gene's expression difference.

Key takeaway: The conditional exact test eliminates the unknown common mean by conditioning on the total count, yielding a beta-binomial null distribution for a simple two-group negative binomial comparison. Its exactness is conditional on its assumptions; the GLM coefficient test targets a model coefficient and can accommodate a broader design.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

Under the conditional test, what does conditioning on $T$ remove from the null distribution?
Two equal-exposure groups each have one sample, dispersion $\phi=0.5$, and total count $t=3$. If group 1 has all three counts, calculate the upper-tail p-value under the null.
Why is a symmetric conditional distribution inappropriate when $m_1=1$ and $m_2=2$?
What quantity does a two-group GLM coefficient with a log exposure offset represent?
Why does plugging an estimated dispersion into the exact-test formula weaken the strict exactness claim?

Solutions. (1) Conditioning on the total removes the unknown common baseline mean $\mu$, leaving a null distribution determined by group sizes and dispersion. (2) Here $r=1/\phi=2$, so $a=b=2$. The upper tail at $S_1=3$ is just $P(S_1=3\mid T=3)=B(5,2)/B(2,2)=(1/30)/(1/6)=0.2$. (3) The beta-binomial parameters are $a=m_1/\phi$ and $b=m_2/\phi$; unequal group sizes give unequal parameters and an asymmetric allocation distribution. (4) It represents the log rate ratio between the groups, conditional on other model terms. (5) The stated conditional distribution treats dispersion as fixed; estimating it introduces uncertainty not included in that reference distribution, so the resulting p-value is not strictly exact.