Fisher's Exact Test for Small Sample Sizes

Introduction

Clinical trials frequently compare two groups on a binary outcome: response versus non-response, treatment success versus failure, adverse event versus no adverse event, or disease present versus absent.

These data are commonly summarized using a 2×2 contingency table. When sample sizes are large, a Pearson chi-square test is often appropriate. But when the sample size is small, or when one or more expected cell counts are small, the usual chi-square approximation may be unreliable.

Fisher's exact test provides an alternative. Rather than relying on an asymptotic chi-square distribution, Fisher's test calculates probabilities directly from the observed margins of the contingency table.

When Is Fisher's Exact Test Used?

Fisher's exact test is especially useful when the data form a 2×2 contingency table and the sample size is small.

Examples in clinical research include:

• A small Phase I or Phase II study comparing response rates
• A rare adverse event occurring in only a few patients
• A pilot study with very few participants
• A small diagnostic study with sparse cells
• A small case-control study
• A subgroup analysis with limited observations

There is no universal sample-size cutoff that says Fisher's test must be used. The important issue is whether the approximation underlying the chi-square test is adequate for the observed data and the analysis objective.

The 2×2 Contingency Table

Suppose a clinical trial compares an experimental treatment with a control. The binary outcome is response.

The total sample size is:

The treatment response rate is:

and the control response rate is:

The question is whether the observed difference in response is compatible with the null hypothesis of no association between treatment assignment and response.

The Null Hypothesis

For a 2×2 table, Fisher's exact test is commonly expressed as a test of independence.

The null hypothesis is:

For a treatment-response example, this can equivalently be expressed as:

The alternative may be two-sided:

or directional. For example, if the scientific question is whether the experimental treatment has a higher response rate:

Why the Chi-Square Approximation Can Be Problematic

The Pearson chi-square test uses an asymptotic approximation. Its test statistic is approximately chi-square distributed when the sample size is sufficiently large.

With sparse data, however, the approximation can be poor.

Consider a table in which several cells contain zero, one, or two observations. The possible tables are highly discrete, and a continuous chi-square distribution may not accurately represent the exact sampling distribution.

Fisher's approach avoids this approximation by calculating probabilities from the exact distribution implied by the table margins.

The Key Idea: Fixed Margins

Fisher's exact test conditions on the marginal totals. For the general table:

Once the margins are fixed, the number of treatment patients who respond is a hypergeometric random variable.

This is the mathematical foundation of Fisher's exact test.

The Hypergeometric Distribution

Suppose:

• \(n\) = total number of patients
• \(K\) = total number of responses
• \(n_T\) = number assigned to treatment
• \(X\) = number of responders assigned to treatment

Then, conditional on the margins:

The probability of observing exactly \(x\) treatment responders is:

This formula is the heart of Fisher's exact test.

What Does the Hypergeometric Formula Mean?

The numerator counts the number of ways to select:

• \(x\) responders for the treatment group, and
• \(n_T-x\) nonresponders for the treatment group.

The denominator counts all possible ways to select \(n_T\) patients from the full population of \(n\) patients.

Therefore, the ratio gives the probability of obtaining exactly \(x\) treatment responders given the fixed row and column totals.

A Complete Worked Clinical Example

Suppose a small randomized clinical study compares an experimental treatment with control. The primary binary outcome is clinical response. The study contains only 20 patients, with 10 assigned to each group.

The observed results are:

The observed response rates are:

and:

The observed difference in response rates is therefore:

or 60 percentage points.

Step 1: Identify the Margins

The table has:

• 20 total patients
• 10 treatment patients
• 10 control patients
• 8 total responders
• 12 total nonresponders

Under Fisher's exact framework, these margins are treated as fixed for the conditional calculation.

Let \(X\) be the number of responders in the experimental group. The observed value is:

Step 2: Determine the Possible Values of \(X\)

Because there are only 8 responders in total and 10 patients in the treatment group, the treatment group could have received between 0 and 8 of the responders.

Thus:

Each possible value corresponds to a different 2×2 table with the same marginal totals.

Step 3: Calculate the Probability of the Observed Table

For the observed value \(X=7\):

The individual components are:

and:

while:

Therefore:

So the probability of obtaining exactly this table, conditional on the marginal totals, is approximately 0.00947.

Step 4: What Does "Exact" Testing Actually Do?

Fisher's test does not usually stop after calculating the probability of the observed table.

For a one-sided test asking whether the treatment response rate is higher, we consider tables at least as favorable to the treatment as the observed table.

Here the observed treatment group has 7 of the 8 total responses. The more extreme tables in the direction of treatment benefit include:

as well as the observed:

Therefore, the one-sided p-value is:

Step 5: Calculate the One-Sided p-Value

We already calculated:

Now calculate the probability of \(X=8\):

Since:

we obtain:

Therefore:

Thus the one-sided Fisher exact p-value is approximately:

At a conventional one-sided significance level of 0.05, this provides evidence against the null hypothesis in favor of a higher response rate in the experimental group.

Two-Sided Fisher's Exact Test

A two-sided test asks whether the two groups differ in either direction. The alternative is:

The two-sided p-value is more complicated than simply doubling a one-sided Fisher p-value.

A commonly used definition sums the probabilities of all tables that are at least as unlikely as the observed table under the null distribution.

For the worked example, the two-sided Fisher p-value is approximately:

This is smaller than 0.05, so the data provide evidence of an association between treatment group and response.

One-Sided vs. Two-Sided Testing

In most confirmatory clinical trials, the directionality of the hypothesis and the significance level should be specified prospectively.

The Odds Ratio

Fisher's exact test answers a hypothesis-testing question, but it does not by itself describe the magnitude of the association.

For a 2×2 table:

the cross-product odds ratio is:

For the worked example:

Thus the estimated odds of response are approximately 21 times as high in the experimental group as in the control group.

Response Rates, Risk Ratio, and Odds Ratio

For the worked example:

The risk ratio is:

The odds in the treatment group are:

The odds in the control group are:

Therefore:

Fisher's Test Is a Test, Not an Effect Estimate

A common reporting mistake is to report only:

"Fisher's exact test was significant, \(p=0.0198\)."

That does not tell the reader how large the treatment difference was.

A stronger clinical report provides:

• Response counts
• Response percentages
• An effect estimate such as an odds ratio or risk ratio
• A confidence interval when appropriate
• The exact p-value

The p-value addresses compatibility with the null hypothesis; the effect estimate addresses the magnitude of the observed association.

Confidence Intervals for the Odds Ratio

For small samples, the odds-ratio confidence interval can be very wide. This is not a flaw in Fisher's test. It reflects the limited information available in a small study.

In the worked example, the point estimate is:

But the uncertainty around an odds ratio estimated from only 20 patients can be substantial.

Why Zero Cells Matter

Small clinical datasets often contain zero counts. For example:

The usual cross-product odds ratio is:

This can create problems for methods based on the logarithm of the odds ratio, because:

is undefined.

Fisher's exact test itself can still be calculated.

This is one reason exact tests are particularly useful for sparse categorical data.

Fisher's Test Does Not Require Large Expected Counts

The chi-square test typically relies on expected counts being sufficiently large for its asymptotic approximation. Fisher's exact test does not make the same large-sample approximation.

For this reason, a table containing expected counts well below 5 can still be an appropriate setting for Fisher's exact test.

Fisher's Test and Randomization

There is an intuitive way to understand the exact test. Imagine that the 20 patients in the worked example are fixed, including their 8 responders and 12 nonresponders. Under the null hypothesis, treatment assignment is unrelated to response. If 10 of the 20 patients are assigned to treatment, different allocations of the 8 responders across the treatment and control groups are possible.

Fisher's test asks: How unusual is the observed allocation of responders?

The hypergeometric distribution provides the exact answer.

The Possible Tables in the Worked Example

Because the margins are fixed, the entire probability distribution can be listed.

The observed study corresponds to \(X=7\). For a directional test favoring treatment, the relevant tail is:

which contains the observed table and the even more treatment-favorable \(X=8\) table.

Why Fisher's Test Is Especially Useful in Small Phase II Studies

Early clinical development frequently involves relatively small patient populations. Suppose a Phase II pilot study enrolls only 12 or 20 patients. A response comparison might look like:

The table is highly sparse. An asymptotic approximation may not provide a reliable description of the sampling distribution, while Fisher's exact test remains directly applicable.

Fisher's Exact Test vs. Pearson Chi-Square

Modern computers make Fisher's test computationally straightforward even for many practical applications.

Does Fisher's Exact Test Always Give a More Conservative Result?

Fisher's exact test is sometimes described as "conservative." That description should be used carefully.

The discreteness of the exact distribution can make attainable p-values coarse, particularly in very small samples. For example, a p-value may jump from a value below 0.05 to a value above 0.05 because only a limited number of possible tables exist.

However, the exact test is not simply a universally conservative replacement for chi-square testing. The results depend on:

• The observed margins
• The alternative hypothesis
• The definition of the exact tail
• The chosen test statistic or ordering rule

Two-Sided p-Values Require Special Care

One of the most common sources of confusion is the two-sided Fisher p-value. For a normal or t distribution, doubling a one-sided tail probability is often straightforward. For a discrete hypergeometric distribution, there is no single universally obvious way to define "equally extreme" tables.

One widely used definition is the probability-ordering approach:

where \(T\) denotes a possible table.

Other exact-test definitions are possible.

Fisher's Test Is Conditional

A subtle but important point is that Fisher's exact test conditions on the marginal totals. This means the test is not simply asking whether two independently estimated binomial proportions differ without qualification.

Instead, it evaluates the distribution of one cell count conditional on the observed margins.

This conditional perspective is particularly natural in randomized experiments, where treatment-group sizes are fixed by design.

Fisher's Test and Randomized Clinical Trials

Suppose a randomized trial assigns exactly 10 patients to each treatment group. The treatment-group total is therefore fixed by design. If the total number of observed responders is also conditioned upon, the hypergeometric distribution describes how those responders could have been distributed between the two treatment groups under the null hypothesis.

This provides an intuitive connection between randomization and Fisher's exact test.

R Implementation

In R, Fisher's exact test is performed using the built-in fisher.test() function.

First create the 2×2 table:

tab <- matrix(
  c(7, 3,
    1, 9),
  nrow = 2,
  byrow = TRUE
)

dimnames(tab) <- list(
  Treatment = c("Experimental", "Control"),
  Outcome = c("Response", "No Response")
)

tab

The resulting table is:

            Outcome
Treatment    Response No Response
Experimental        7           3
Control             1           9

Two-Sided Fisher's Exact Test in R

fisher.test(
  tab,
  alternative = "two.sided"
)

The resulting two-sided p-value is approximately:

p-value
0.01977

The odds ratio estimate is:

odds ratio
21

R also provides an exact confidence interval for the odds ratio.

One-Sided Fisher's Exact Test in R

If the prespecified alternative is that the experimental treatment has a higher response rate:

fisher.test(
  tab,
  alternative = "greater"
)

For this table, the one-sided p-value is approximately:

p-value
0.00983

The direction of the alternative depends on the ordering of the table. Here the experimental group is the first row, so greater corresponds to greater odds of response in the experimental group.

Extracting the Odds Ratio and p-Value

result <- fisher.test(
  tab,
  alternative = "two.sided"
)

result$estimate
result$p.value
result$conf.int

This makes it easy to incorporate Fisher's test into an analysis pipeline.

Calculating the Hypergeometric Probability in R

The exact probability underlying Fisher's test can also be calculated using R's hypergeometric distribution functions.

For the worked example:

• Total population: \(N=20\)
• Total responders: \(K=8\)
• Treatment-group size: \(n_T=10\)
• Observed treatment responders: \(x=7\)

The exact probability is:

dhyper(
  7,
  m = 8,
  n = 12,
  k = 10
)

which gives approximately:

0.009472

This is the probability of exactly the observed \(X=7\) table.

Calculating the One-Sided Tail in R

The probability of observing 7 or more treatment responders is:

phyper(
  6,
  m = 8,
  n = 12,
  k = 10,
  lower.tail = FALSE
)

This gives approximately:

0.009829

which matches the one-sided Fisher exact p-value.

Building the Exact Distribution

It can be instructive to calculate all possible hypergeometric probabilities.

x <- 0:8

prob <- dhyper(
  x,
  m = 8,
  n = 12,
  k = 10
)

data.frame(
  treatment_responders = x,
  probability = prob
)

The probabilities sum to one:

sum(prob)
# 1

This is the complete conditional null distribution for the number of treatment responders given the margins.

Why the Exact Distribution Is Discrete

Unlike a normal distribution, the hypergeometric distribution can only take integer values. For the worked example:

There is no possible value such as:

This discreteness is why exact p-values can behave differently from large-sample approximations.

Fisher's Exact Test With Very Small Samples

Consider an even smaller study:

Here, the sample size is only 10 and one cell is zero. An exact method is particularly natural.

The Fisher two-sided p-value for this table is approximately:

The exact test can therefore be performed without relying on a large-sample chi-square approximation.

Fisher's Test and Rare Adverse Events

Fisher's exact test is also common when comparing rare safety events. For example:

Even though the overall sample contains 60 patients, the event counts are small. The sparse-event structure can make an exact analysis attractive, particularly when the primary question concerns a simple 2×2 comparison.

Fisher's Test for Diagnostic Studies

The same framework applies to diagnostic accuracy studies. For example:

A Fisher exact test can assess association between test result and disease status when the sample is small. However, diagnostic studies usually require additional measures such as:

• Sensitivity
• Specificity
• Positive predictive value
• Negative predictive value
• Likelihood ratios
• Confidence intervals

The Fisher p-value should therefore be viewed as only one component of the analysis.

Fisher's Test Does Not Measure Clinical Importance

Suppose a very small trial produces a large response difference but a non-significant p-value. That does not necessarily mean the treatment difference is clinically unimportant.

It may simply mean the study does not contain enough information to establish the effect with the desired statistical certainty.

Conversely, in a very large study, a tiny treatment difference can produce a very small p-value.

Fisher's Test vs. Barnard's Exact Test

Fisher's exact test is not the only exact method for a 2×2 table. Another approach is Barnard's exact test.

The conceptual distinction is important. Fisher's test conditions on both margins, whereas Barnard's test is an unconditional exact test and does not condition on the total number of events in the same way.

Fisher's test remains extremely common because it is conceptually simple, well established, and widely implemented.

Fisher's Test vs. Chi-Square With Continuity Correction

Another common alternative for sparse 2×2 tables is the chi-square test with Yates' continuity correction. The correction attempts to improve the approximation by accounting for the discreteness of the data.

However, it remains an asymptotic method. Fisher's test instead calculates probabilities from the exact conditional distribution.

Common Mistake: Automatically Using Fisher for Every 2×2 Table

Fisher's exact test is useful, but it does not need to replace every chi-square analysis.

For large, well-populated tables, Pearson's chi-square test can be entirely appropriate and computationally simpler.

The important principle is:

Common Mistake: Calling Fisher "Nonparametric"

Fisher's exact test is sometimes casually labeled a nonparametric test. That terminology is not particularly informative.

The more useful description is that Fisher's test is an exact conditional test for a 2×2 contingency table.

It does not require the normality assumption associated with many parametric methods, but the term "nonparametric" does not explain the actual probability model.

Common Mistake: Ignoring the Alternative Hypothesis

Suppose the treatment response table is:

If the question is: Is the experimental treatment better? then a one-sided alternative may be scientifically appropriate if prespecified.

If the question is: Are the treatment groups different? then a two-sided test is appropriate.

These are different hypotheses and can produce different p-values.

Common Mistake: Doubling the One-Sided p-Value

For the worked example:

Simply doubling this gives:

which happens to be very close to the commonly reported two-sided value for this example.

But this should not be treated as a general rule for Fisher's exact test.

Because the exact distribution is discrete and asymmetric, the two-sided p-value is determined by the specified exact-test ordering rule.

Common Mistake: Reporting Only the p-Value

A better clinical report might state:

"Response occurred in 7/10 patients (70%) in the experimental group and 1/10 patients (10%) in the control group. The odds ratio was 21.0, and Fisher's exact test gave a two-sided p-value of 0.0198."

This communicates:

• The observed data
• The magnitude of the difference
• The effect estimate
• The statistical evidence

Common Mistake: Confusing Odds With Probability

A 70% response probability corresponds to odds of:

It does not correspond to odds of 0.70.

Likewise, a 10% response probability corresponds to odds:

Therefore the odds ratio is:

Common Mistake: Interpreting the Odds Ratio as a Risk Ratio

In the worked example:

These are not interchangeable.

When the outcome is relatively common, the difference between the odds ratio and risk ratio can be substantial.

Common Mistake: Assuming a Significant Result Proves Causality

Fisher's test detects evidence of association. It does not by itself establish causality.

Causal interpretation depends on the study design. A randomized clinical trial provides a much stronger basis for causal inference than an observational comparison, assuming the trial is properly designed and conducted.

Multiple Fisher Tests

Suppose a clinical trial performs Fisher's exact test for:

• Overall response
• Complete response
• Disease control
• Several safety events
• Several biomarker-defined subgroups

If many hypotheses are tested, the probability of obtaining at least one small p-value by chance increases.

The use of Fisher's exact test does not automatically solve the multiplicity problem.

When multiple confirmatory hypotheses are tested, an appropriate multiplicity strategy should be prespecified.

Fisher's Test in Subgroup Analyses

Fisher's test is sometimes used within small subgroups. For example, a clinical trial might examine response separately among:

• Biomarker-positive patients
• Biomarker-negative patients
• Prior-treatment subgroups
• Age categories
• Disease-stage categories

If a subgroup contains very few patients, Fisher's exact test may be useful for a simple 2×2 comparison.

However, subgroup analyses introduce additional issues:

• Multiple testing
• Low statistical power
• Wide confidence intervals
• Potential confounding
• Interaction testing

A significant result in one subgroup and a non-significant result in another does not automatically demonstrate that the treatment effects differ between subgroups.

Fisher's Test Does Not Test Interaction

Suppose:

Subgroup A: treatment is significant.

Subgroup B: treatment is not significant.

It is tempting to conclude that the treatment works differently between the subgroups. That conclusion is not justified merely by comparing two p-values.

A formal interaction analysis is required to test whether the treatment effect differs between subgroups.

Clinical Trial Reporting Example

A concise results statement could be:

For a confirmatory clinical report, the exact confidence interval for the effect estimate should also be reported, along with the prespecified analysis population and handling of missing data.

Missing Data and Fisher's Exact Test

Fisher's exact test requires a defined 2×2 analysis table. Clinical trials may contain:

• Patients without post-baseline assessments
• Withdrawals
• Protocol deviations
• Missing endpoint measurements
• Patients who discontinue treatment early

The statistical analysis plan should specify how such patients are classified for the binary endpoint.

Possible approaches depend on the endpoint and estimand and may include prespecified nonresponder rules, exclusion under defined circumstances, multiple imputation, or other methods.

Continuity Corrections Are Not Needed for Fisher's Test

Because Fisher's test is based directly on the exact discrete distribution, there is no need to apply a continuity correction such as Yates' correction to the Fisher calculation.

Continuity corrections are associated with approximating a discrete distribution using a continuous distribution such as the chi-square distribution.

Computational Interpretation

Fisher's exact test can be understood algorithmically. Given the observed table:

The Mathematical Core in One Formula

For a table with fixed margins, the probability of observing \(x\) treatment responders is:

where:

• \(N\) = total sample size
• \(K\) = total number of successes
• \(n_T\) = treatment-group sample size
• \(x\) = number of successes in the treatment group

Everything else in Fisher's exact test follows from this distribution and the definition of the test's rejection region.

Worked Example: All Key Quantities

When Fisher's Exact Test Is a Good Choice

Fisher's exact test is particularly attractive when:

• The outcome is binary.
• The exposure or treatment has two categories.
• The resulting data form a 2×2 table.
• The sample size is small.
• One or more cells are sparse.
• Expected counts are small.
• A zero cell is present.
• An exact conditional analysis is scientifically appropriate.

When Fisher's Exact Test May Not Be Enough

A 2×2 Fisher test is not automatically the best analysis for every clinical-trial question. More complex methods may be preferable when:

• There are more than two treatment groups.
• The outcome has more than two categories.
• The endpoint is continuous.
• The endpoint is time-to-event.
• Covariate adjustment is required.
• Repeated observations are available.
• There are multiple strata that should be modeled.
• The treatment effect must be adjusted for baseline prognostic factors.

For example, logistic regression may be more appropriate when adjustment for important baseline covariates is required.

Fisher's Exact Test and Logistic Regression

For a simple unadjusted 2×2 analysis, Fisher's exact test can be an excellent choice for sparse data. But if the analysis needs adjustment for age, disease severity, biomarker status, or another prognostic factor, logistic regression may provide a more appropriate framework.

The basic logistic regression model is:

The exponentiated treatment coefficient:

is interpreted as an adjusted odds ratio under the specified model.

Fisher's Test and Stratified Data

Suppose a clinical trial has several centers, and the treatment-response relationship must be evaluated while accounting for center. Simply pooling all patients into one 2×2 table may ignore important stratification.

Methods such as the Cochran-Mantel-Haenszel approach can be considered for stratified categorical data.

The appropriate method depends on the design and estimand.

Fisher's Test and Matched Data

If the data are paired or matched rather than independent, a standard 2×2 Fisher test may not be appropriate. For paired binary observations, McNemar's test is a commonly considered alternative.

This illustrates an important principle: the data-generating design matters as much as the observed cell counts.

A Practical Analysis Workflow

What Should Be Reported?

A clear clinical-trial report should generally include:

• The 2×2 cell counts
• The response percentage in each group
• The effect estimate
• A confidence interval where appropriate
• The exact p-value
• Whether the test was one-sided or two-sided
• The statistical method used
• The analysis population
• Any prespecified handling of missing data

For example:

Fisher's Exact Test in SAS

Fisher's exact test is also straightforward in SAS using PROC FREQ.

data response;
    input treatment $ response $ count;
    datalines;
Experimental Yes 7
Experimental No  3
Control      Yes 1
Control      No  9
;
run;

proc freq data=response;
    tables treatment*response / fisher;
    weight count;
run;

The FISHER option requests Fisher's exact test.

If the individual patient records are available rather than aggregated counts, the WEIGHT statement is unnecessary.

SAS With Individual Patient Records

data response;
    input treatment $ response $;
    datalines;
Experimental Yes
Experimental Yes
Experimental Yes
Experimental Yes
Experimental Yes
Experimental Yes
Experimental Yes
Experimental No
Experimental No
Experimental No
Control Yes
Control No
Control No
Control No
Control No
Control No
Control No
Control No
Control No
Control No
;
run;

proc freq data=response;
    tables treatment*response / fisher;
run;

This produces the 2×2 table together with the Fisher exact test.

R vs. SAS

Interpretation of the Worked Example

The experimental treatment produced 7 responses among 10 patients, compared with only 1 response among 10 control patients.

The response-rate difference was:

The estimated odds ratio was:

The two-sided Fisher exact p-value was approximately:

Therefore, under the specified exact testing framework, the observed treatment allocation of responders would be relatively unusual under the null hypothesis of no association.

However, the study contains only 20 patients, so the estimated treatment effect should be interpreted with appropriate attention to statistical uncertainty.

The Most Important Concept

The central idea behind Fisher's exact test is simple: condition on the observed margins and calculate the exact probability of the possible 2×2 tables.

For a binary outcome and two groups, the hypergeometric distribution describes how the total successes could be distributed between the groups under the null hypothesis.

The p-value is then obtained by summing the probabilities of the observed table and the tables considered at least as extreme according to the selected alternative and exact-test definition.

This makes Fisher's exact test particularly useful for sparse clinical data.

References

Fisher, R.A. (1922). On the interpretation of χ² from contingency tables, and the calculation of P. Journal of the Royal Statistical Society, 85(1), 87–94.
Fisher, R.A. (1934). Statistical Methods for Research Workers. Oliver and Boyd.
Agresti, A. (2013). Categorical Data Analysis, 3rd ed. Wiley.
Mehta, C.R. & Patel, N.R. (1983). A network algorithm for performing Fisher's exact test in r×c contingency tables. Journal of the American Statistical Association, 78(382), 427–434.
Mehta, C.R. & Patel, N.R. (1986). Algorithm 643: FEXACT, a FORTRAN subroutine for Fisher's exact test on unordered r×c contingency tables. ACM Transactions on Mathematical Software, 12(2), 154–161.
SAS Institute Inc. SAS/STAT User's Guide: The FREQ Procedure.
R Core Team. R: A Language and Environment for Statistical Computing.