Defining Two-Sided Exact Binomial P-Values

Why a Two-Sided Exact P-Value Needs a Convention

For a one-sided test, the direction of evidence identifies which counts are more extreme: unusually few successes or unusually many. In a two-sided test, outcomes can be unusual in either direction. With a discrete binomial count, however, there is not always a unique way to pair an outcome in one tail with an equally extreme outcome in the other. The two tails can also have very different probabilities.

This means that “two-sided exact p-value” does not specify a unique calculation by itself. This tutorial defines one standard approach, called probability ordering: include every possible count whose probability under the null is no greater than the probability of the observed count, then add those probabilities. The rule is exact for the stated binomial model, but its result can differ from other two-sided conventions.

The Binomial Model and Probability-Ordering Definition

Suppose a fixed number $n$ of independent trials each has the same probability $p$ of success, and the recorded outcome on each trial is success or failure. Let $X$ be the total number of successes. Under a null hypothesis $H_0:p=p_0$, the model is $X\sim\operatorname{Binomial}(n,p_0)$. The probability of exactly $k$ successes under this null is

f_0(k)=P(X=k\mid H_0)=\binom{n}{k}p_0^k(1-p_0)^{n-k}, \qquad k=0,1,\ldots,n.

For a two-sided test, the hypotheses are $H_0:p=p_0$ and $H_a:p\ne p_0$. Write $x_{\text{obs}}$ for the observed number of successes. Probability ordering compares the null probability of each possible count with the null probability of the observed count. It does not rank counts by their distance from $np_0$.

Definition: The probability-ordering two-sided exact binomial p-value is the sum of the null probabilities of all possible counts that are at least as unlikely as the observed count: $$ p_{\text{ord}}=\sum_{\substack{k=0,\ldots,n\\ f_0(k)\le f_0(x_{\text{obs}})}} f_0(k). $$ The comparison includes equality. Thus, if another count has exactly the same null probability as the observed count, it is included.

The model assumptions matter: the number of trials is fixed, each trial has two possible outcomes, the trials are independent, and each has the same success probability under the null. The binomial calculation is exact when this model describes the data. “Exact” refers to using the binomial probabilities rather than a continuous approximation; it does not mean that every possible two-sided convention gives the same answer.

Worked Example: A Symmetric Null With Ten Trials

Suppose a hypothetical process is checked on $n=10$ independent items. Each item meets a binary criterion with null probability $p_0=0.5$, and $x_{\text{obs}}=8$ items meet it. Assume the number checked was fixed, each item is classified as meeting or not meeting the criterion, and the items have the same independent success probability under $H_0$. Find the probability-ordering p-value for $H_0:p=0.5$ versus $H_a:p\ne0.5$.

Under the null, $X\sim\operatorname{Binomial}(10,0.5)$. The probability of the observed count is

f_0(8)=\binom{10}{8}(0.5)^8(0.5)^2 =\frac{45}{1024} \approx 0.04395.

The possible counts with probability no greater than $45/1024$ are $0,1,2,8,9,$ and $10$. For example, $f_0(2)=45/1024$, so count 2 is included by the equality rule; counts 3 through 7 have larger probabilities. Therefore,

p_{\text{ord}} =f_0(0)+f_0(1)+f_0(2)+f_0(8)+f_0(9)+f_0(10) =\frac{1+10+45+45+10+1}{1024} =\frac{112}{1024} =0.1094.

The probability-ordering p-value is about 0.1094. It is the probability, assuming $p=0.5$, of obtaining one of the counts in this specified set. It is not the probability that $H_0$ is true.

How the Convention Can Change the Answer

A frequently used alternative is the doubled-smaller-tail convention: calculate the smaller of the two inclusive tail probabilities, double it, and cap the result at 1. In symbols, this convention gives

p_{\text{double}} =\min\left\{1,\;2\min\big(P(X\le x_{\text{obs}}\mid H_0),P(X\ge x_{\text{obs}}\mid H_0)\big)\right\}.

For the symmetric null in the preceding example, this happens to agree with probability ordering. The binomial distribution is symmetric around 5, so the counts in the two tails pair neatly. For an asymmetric null, the conventions need not agree.

Worked Example: An Asymmetric Null and Different Two-Sided Values

Suppose a hypothetical set of $n=10$ independent trials has null success probability $p_0=0.2$, with $x_{\text{obs}}=3$. Assume a fixed number of binary trials, a common success probability, and independence. Test $H_0:p=0.2$ against $H_a:p\ne0.2$ using probability ordering, then compare with the doubled-smaller-tail convention.

The observed count has null probability

f_0(3)=\binom{10}{3}(0.2)^3(0.8)^7 \approx 0.201327.

The null probabilities for counts 0, 1, and 2 are approximately 0.107374, 0.268435, and 0.301990. Thus count 0 is no more likely than the observed count, while counts 1 and 2 are more likely. Count 3 itself has probability equal to the comparison value, and every count from 4 through 10 has a smaller probability. The probability-ordering set is therefore $\{0,3,4,5,6,7,8,9,10\}$. Its total probability is

p_{\text{ord}}=P(X=0\mid H_0)+P(X\ge3\mid H_0) \approx 0.107374+0.322200 =0.4296.

For the alternative convention, the inclusive lower-tail probability is $P(X\le3\mid H_0)\approx0.879126$, and the inclusive upper-tail probability is $P(X\ge3\mid H_0)\approx0.322200$. The smaller tail is the upper tail, so

p_{\text{double}}=2(0.322200)\approx0.6444.

The two conventions give different values, 0.4296 and 0.6444, for the same data and null model. Probability ordering includes count 0 because its individual probability is no greater than the observed count’s probability. The doubled-tail rule instead doubles the smaller inclusive tail. Neither calculation changes the data or the null model; they differ in how “at least as extreme” is defined for a discrete two-sided test.

Worked Example: Ordering Outcomes When Zero Successes Are Observed

In a hypothetical experiment, $n=8$ independent binary trials are conducted under a null success probability $p_0=0.25$, and no successes are observed, so $x_{\text{obs}}=0$. Assume that the trial count is fixed and that the trials share the same independent success probability under the null. Calculate the probability-ordering p-value.

Under $H_0$, $X\sim\operatorname{Binomial}(8,0.25)$, and the observed count has probability

f_0(0)=\binom{8}{0}(0.25)^0(0.75)^8 =\frac{6561}{65536} \approx0.100113.

Count 1 has probability $17496/65536$, count 2 has probability $20412/65536$, and count 3 has probability $13608/65536$; each exceeds the observed probability. Count 4 has probability $5670/65536$, which is smaller, and counts 5 through 8 have still smaller probabilities. The probability-ordering set is $\{0,4,5,6,7,8\}$, giving

p_{\text{ord}} =\frac{6561+5670+1512+252+24+1}{65536} =\frac{14020}{65536} \approx0.2139.

This example shows why the probability-ordering set need not consist of two equal-length tails around the observed count. It contains the observed zero-success outcome and the upper counts whose individual null probabilities are no greater than its probability, but it excludes counts 1, 2, and 3.

Why the Probability-Ordering Rule Is a Valid Exact Test

The rule can be justified directly from the finite set of possible binomial outcomes. Group counts that have equal null probability. List the distinct probability levels from smallest to largest, and let $w_j$ be the total null probability of all counts at level $j$. Define $C_j=w_1+\cdots+w_j$, the cumulative probability through level $j$. If the observed count has probability level $j$, the probability-ordering p-value is exactly $C_j$, because it includes all outcomes at that level and all lower-probability levels.

Now choose a significance level $\alpha$, and reject $H_0$ when the p-value is at most $\alpha$. The outcomes that lead to rejection are the complete probability groups for which $C_j\le\alpha$. If there is at least one such group, the null probability of rejection is the largest such cumulative total, which is no greater than $\alpha$. If there is no such group, the rejection probability is zero. Thus, under the null, the probability of rejection is at most the stated significance level.

This may be strictly less than $\alpha$, because counts come in whole-number steps and probability groups cannot generally be split. That is the sense in which a discrete exact test can be conservative: its actual null rejection probability may be below its nominal significance level. Exactness does not require the rejection probability to equal $\alpha$.

Using and Reporting the Result

A two-sided exact p-value should be reported with the convention used. For example: “Using probability ordering, the exact binomial p-value is 0.4296.” If a software package uses a different two-sided convention, its result can differ without either computation being an arithmetic error. To compare reported results, identify the null probability model and the rule used to define outcomes at least as extreme as the observation.

At significance level $\alpha$, compare the selected p-value with $\alpha$. A p-value no greater than $\alpha$ is evidence against $H_0$ according to this test; a larger p-value does not establish that $H_0$ is true. The p-value is calculated under the null model and summarizes the probability of the designated set of outcomes, not the probability that the null hypothesis is correct.

Common Mistakes

Assuming “two-sided” dictates one calculation. Discrete counts do not generally have a unique opposite-tail match. State whether the calculation uses probability ordering or another convention.
Summing only counts at least as far from $np_0$ as the observed count. Distance from the expected count is not the probability-ordering rule. Compare each count’s null probability with $f_0(x_{\text{obs}})$.
Leaving out equal-probability outcomes. The probability-ordering definition uses $f_0(k)\le f_0(x_{\text{obs}})$, so ties must be included.
Calling an exact p-value the chance the null is true. It is a null-model probability for a specified set of possible outcomes, not a probability assigned to a hypothesis.

Key takeaway: For probability ordering, calculate the null probability of the observed count, find every binomial outcome with probability no greater than that value, and sum their probabilities. Because binomial outcomes are discrete, a two-sided exact p-value depends on the stated extremeness convention.

Check Your Understanding

Use the probability-ordering definition unless a question asks for another convention.

For $n=12$, $p_0=0.5$, and $x_{\text{obs}}=10$, write the expression that defines the probability-ordering p-value. Which counts must be compared with the observed count’s null probability?
Why does the probability-ordering rule include all counts tied with the observed count’s null probability?
In the example with $n=10$, $p_0=0.2$, and $x_{\text{obs}}=3$, why do probability ordering and the doubled-smaller-tail convention give different values?
State the binomial model assumptions needed for a one-proportion exact test.
Why can an exact discrete test have a null rejection probability smaller than its significance level?