Fisher's Exact Test for Two-by-Two Tables

Why Fisher’s Exact Test Conditions on the Margins

A two-by-two table can record the relationship between a binary group indicator and a binary outcome. When counts are small, large-sample approximations for testing independence may be inaccurate. Fisher’s exact test instead calculates probabilities from the conditional distribution of a table cell count, given the observed row and column totals.

Conditioning on the margins is central: once those totals are fixed, every possible table is determined by one cell. Under the null hypothesis of no association, that cell has a hypergeometric distribution. The test’s p-value is then a tail probability, or, for a common two-sided definition, a sum over tables at least as improbable under the null as the observed table.

The Conditional Distribution

Write the observed table and its margins as follows. The first row might represent a treatment group, and the first column a binary outcome of interest.

\begin{array}{c|cc|c} & \text{Outcome 1} & \text{Outcome 0} & \text{Row total}\\ \hline \text{Group 1} & a & b & r_1\\ \text{Group 0} & c & d & r_2\\ \hline \text{Column total} & c_1 & c_2 & n \end{array} \qquad r_1=a+b,\quad r_2=c+d,\quad c_1=a+c,\quad c_2=b+d,\quad n=r_1+r_2.

Let $X$ be the count in the first row and first column, so its observed value is $a$. Once the margins are fixed, a possible value $x$ determines all four cells: $x,\ r_1-x,\ c_1-x,$ and $r_2-c_1+x$. Each must be nonnegative.

Definition: With fixed row totals $r_1,r_2$ and column totals $c_1,c_2$, the support of $X$ is $x_{\min}\leq X\leq x_{\max}$, where $x_{\min}=\max(0,r_1-c_2)$ and $x_{\max}=\min(r_1,c_1)$. Under the null hypothesis of independence, its conditional probability mass function is $$ P(X=x\mid\text{margins}) = \frac{\binom{c_1}{x}\binom{c_2}{r_1-x}}{\binom{n}{r_1}}, \qquad x_{\min}\leq x\leq x_{\max}. $$ This is a hypergeometric distribution: $r_1$ objects are selected from $n$, of which $c_1$ are in the first column and $c_2$ in the second.

The support bounds follow directly from the cell counts. The cells $x$ and $r_1-x$ require $x\geq0$ and $x\leq r_1$; the cells $c_1-x$ and $r_2-c_1+x$ require $x\leq c_1$ and $x\geq c_1-r_2=r_1-c_2$. Combining these inequalities gives the stated bounds.

Result (conditional hypergeometric law): Suppose the two groups have fixed sizes $r_1,r_2$, and under the null each unit independently has the outcome of interest with the same probability $\pi$, where $0<\pi<1$. Conditional on the observed total $c_1$ of outcome-1 units, the $c_1$ units with that outcome occupy a uniformly distributed subset of size $c_1$ among the $n$ units. Consequently, the count $X$ in group 1 has the hypergeometric probability mass function above.

Proof. Under the null, any particular set of $c_1$ units having outcome 1 has probability $\pi^{c_1}(1-\pi)^{n-c_1}$. This probability is the same for every such set. Given the total $c_1$, all $\binom{n}{c_1}$ sets are therefore equally likely. For $X=x$, choose $x$ of the $c_1$ outcome-1 units from group 1 and $c_1-x$ from group 2. There are $\binom{r_1}{x}\binom{r_2}{c_1-x}$ such assignments, out of $\binom{n}{c_1}$. This gives a hypergeometric probability. It is algebraically the same as the formula in the definition: by factorial expansion,

\frac{\binom{r_1}{x}\binom{r_2}{c_1-x}}{\binom{n}{c_1}} = \frac{\binom{c_1}{x}\binom{c_2}{r_1-x}}{\binom{n}{r_1}},

since $r_2-c_1+x=c_2-r_1+x$ and $n=r_1+r_2=c_1+c_2$. The right-hand expression also has a direct selection interpretation: choose the $r_1$ group-1 units from the $n$ units, with $x$ from the $c_1$ outcome-1 units and $r_1-x$ from the $c_2$ outcome-0 units. This proves the result. $\square$

One-Sided and Two-Sided P-Values

With the first row and first column designated as above, a large $X$ means that group 1 has more outcome-1 units than would typically be expected under independence, given the margins. A small $X$ means fewer. The directional p-values are the corresponding conditional tail probabilities.

Definition: For observed value $a$, the upper-tail Fisher p-value is $P(X\geq a\mid\text{margins})$, for an alternative in the direction of larger $X$. The lower-tail p-value is $P(X\leq a\mid\text{margins})$, for an alternative in the direction of smaller $X$. A common two-sided Fisher p-value is the probability-ordering value $$ \sum_{\substack{x_{\min}\leq x\leq x_{\max}\\ P(X=x\mid\text{margins})\leq P(X=a\mid\text{margins})}} P(X=x\mid\text{margins}). $$

The probability-ordering definition includes the observed table and every possible table whose individual null probability is no greater than the observed table’s probability. All ties are included. This definition is widely used, but “two-sided exact p-value” does not specify a unique convention in every setting; another convention can give a different result. A reported analysis should state which definition it uses.

Worked Example: A Small Table and a Directional Test

Suppose the observed table is

\begin{array}{c|cc|c} & \text{Outcome 1} & \text{Outcome 0} & \text{Total}\\ \hline \text{Group 1} & 3 & 0 & 3\\ \text{Group 0} & 0 & 3 & 3\\ \hline \text{Total} & 3 & 3 & 6 \end{array}

Here $r_1=3,\ c_1=3,\ c_2=3,\ n=6$, and the observed count is $a=3$. The support is $0\leq X\leq3$. The probability of each value is

P(X=x\mid\text{margins}) =\frac{\binom{3}{x}\binom{3}{3-x}}{\binom{6}{3}}, \qquad \big(P(X=0),P(X=1),P(X=2),P(X=3)\big) =\left(\frac1{20},\frac9{20},\frac9{20},\frac1{20}\right).

For the upper-tail alternative, the only support value at least as large as the observed value is 3, so $p_{\text{upper}}=1/20=0.05$. For the lower-tail alternative, every support value is at most 3, so $p_{\text{lower}}=20/20=1$. The observed probability is $1/20$; both $x=0$ and $x=3$ have probability no greater than it. Thus the probability-ordered two-sided value is $2/20=0.10$.

Worked Example: Probability-Ordered Two-Sided P-Value

Consider the observed table

\begin{array}{c|cc|c} & \text{Outcome 1} & \text{Outcome 0} & \text{Total}\\ \hline \text{Group 1} & 0 & 3 & 3\\ \text{Group 0} & 4 & 2 & 6\\ \hline \text{Total} & 4 & 5 & 9 \end{array}

The margins give $r_1=3,\ c_1=4,\ c_2=5,\ n=9$, with $a=0$. The support is $0\leq X\leq3$, and $\binom{9}{3}=84$. Applying the hypergeometric formula at each support value gives

\begin{array}{c|cccc} x&0&1&2&3\\ \hline \binom{4}{x}\binom{5}{3-x}&10&40&30&4\\ P(X=x\mid\text{margins})&10/84&40/84&30/84&4/84 \end{array}

For the lower-tail alternative, the observed value is the smallest support value, so $p_{\text{lower}}=P(X=0)=10/84=5/42$. For the upper-tail alternative, all support values are at least 0, so $p_{\text{upper}}=1$.

For the probability-ordered two-sided value, compare each probability with the observed probability $10/84$. The observed value $x=0$ is included, since its probability equals $10/84$. The only other support value with probability no greater than the observed probability is $x=3$, whose probability is $4/84$. The values $x=1$ and $x=2$ have probabilities $40/84$ and $30/84$, both greater than $10/84$. Therefore

p_{\text{two-sided}}=\frac{10}{84}+\frac{4}{84} =\frac{14}{84}=\frac16.

Exactness and Discreteness

The word “exact” refers to using the finite-sample conditional distribution rather than an approximation to it. It does not mean that a p-value must equal a chosen significance level, nor that every test has probability exactly $\alpha$ of rejection under the null. Because $X$ has a discrete distribution, attainable tail probabilities generally jump over many values between 0 and 1.

Result (conditional validity): For a fixed set of margins satisfying the null model, the upper-tail p-value $p(X)=P(X'\geq X\mid\text{margins})$, where $X'$ has the same conditional null distribution as $X$, obeys $P(p(X)\leq\alpha\mid\text{margins})\leq\alpha$ for every $0\leq\alpha\leq1$. The same inequality holds for a lower-tail p-value and for the probability-ordered two-sided p-value defined above.

Proof. For an upper-tail test, list the support values in increasing order, $x_1<\cdots<x_m$, and let $q_i=P(X=x_i\mid\text{margins})$. The p-value at $x_i$ is $\sum_{j=i}^m q_j$, which decreases as $i$ increases. The set of support points whose p-values are at most $\alpha$ is therefore either empty or a final segment $x_k,\ldots,x_m$. Its probability is $\sum_{j=k}^m q_j$, exactly the p-value at $x_k$, and hence at most $\alpha$. For the lower tail, the corresponding set is an initial segment, with probability equal to the p-value at its final point; this too is at most $\alpha$.

For the probability-ordered value, group support points by their probability masses. Let the distinct masses be $q_1<q_2<\cdots<q_s$, and let $w_j$ be the total probability of all points with mass $q_j$. At a point with mass $q_j$, the probability-ordered p-value is $\sum_{i=1}^j w_i$. These values increase with $j$. Thus the points with p-value at most $\alpha$ consist of the first several mass groups, or none; their total probability is the cumulative sum for the last included group and is at most $\alpha$. This proves the conditional validity claim. $\square$

Worked Example: A More Extreme Upper-Tail Table

Suppose each group has size 4 and each outcome column has total 4. The observed table has all four outcome-1 units in group 1:

\begin{array}{c|cc|c} & \text{Outcome 1} & \text{Outcome 0} & \text{Total}\\ \hline \text{Group 1} & 4 & 0 & 4\\ \text{Group 0} & 0 & 4 & 4\\ \hline \text{Total} & 4 & 4 & 8 \end{array}

Here $a=4$, and the support is $0,1,2,3,4$. Since $\binom{8}{4}=70$, the probabilities are

P(X=x\mid\text{margins}) =\frac{\binom{4}{x}\binom{4}{4-x}}{70}, \qquad \big(P(X=0),\ldots,P(X=4)\big) =\frac1{70}(1,16,36,16,1).

The upper-tail p-value is $P(X\geq4)=1/70$. The lower-tail p-value is 1. For the two-sided probability-ordered value, only $x=0$ and $x=4$ have probability no greater than the observed probability $1/70$, so $p_{\text{two-sided}}=2/70=1/35$. In this symmetric example the two-sided value is twice the upper-tail value; that shortcut does not hold in general.

Interpretation and Common Pitfalls

Fisher’s exact test answers a conditional question: given the observed margins, how much probability does the null distribution assign to tables at least as extreme as the observed one, according to the chosen direction or two-sided convention? The test does not by itself measure the size of an association, and a large p-value does not establish that the groups have identical outcome probabilities.

Using the wrong tail. A claim that group 1 has a higher outcome-1 rate calls for an upper-tail test with this table orientation. If rows or columns are reversed, the corresponding direction must be reconsidered.
Calling every two-sided p-value the same quantity. Probability ordering sums probabilities of tables no more likely than the observed table; it is not automatically twice the smaller one-sided tail.
Excluding a tie. The probability-ordering definition includes tables whose probability equals the observed probability. In the example with $a=0$, the observed table itself is included because $10/84\leq10/84$.
Assuming “exact” means equality to the nominal level. Discrete conditional distributions usually make tests conservative at some levels: the rejection probability is at most, not necessarily equal to, $\alpha$.

Key takeaway: Fix the table orientation and margins, derive the hypergeometric probabilities over the feasible support, and then sum the probabilities specified by the alternative. For a two-sided analysis, state the convention and include all probability ties.

Check Your Understanding

Use the fixed-margin hypergeometric distribution to answer these questions.

For row totals $r_1=5,r_2=7$ and column totals $c_1=4,c_2=8$, what is the support of $X$?
In a table with observed first cell $a=2$, what conditional probability sum defines the upper-tail p-value?
Why is the two-sided probability-ordering p-value not necessarily twice the smaller one-sided p-value?
If two support values have the same probability as the observed table, how does the probability-ordering definition treat them?
What does the conditional validity result guarantee about the probability of obtaining a p-value at most $\alpha$ under the null?