Family-Wise Error Rate vs. False Discovery Rate

Introduction

Suppose a researcher performs one hypothesis test at the conventional significance level of 0.05. The probability of a false positive, assuming the null hypothesis is true, is at most 5%.

Now suppose the researcher performs 20 independent hypothesis tests, each at the same 0.05 significance level. The probability of obtaining at least one false positive can be substantially larger than 5%.

This is the fundamental multiple-testing problem.

Two of the most important ways to describe and control errors arising from multiple testing are the:

• Family-wise error rate (FWER)
• False discovery rate (FDR)

These quantities address different scientific goals. FWER asks: "What is the probability of making even one false rejection?"

FDR asks: "Among the hypotheses I reject, what proportion are expected to be false?"

Why Multiple Testing Is a Problem

Consider a single null hypothesis that is tested at:

If the null hypothesis is true, the probability of rejecting it incorrectly is 5%.

Now suppose there are \(m\) independent true null hypotheses, each tested at the same significance level. The probability of making no false rejection is approximately:

Therefore, the probability of making at least one false rejection is:

For 20 independent tests at \(\alpha=0.05\):

Thus, if all 20 null hypotheses are true, there is approximately a 64.2% probability of obtaining at least one nominally significant result.

What Is a Hypothesis Family?

Both FWER and FDR require a definition of the set of hypotheses being considered. This set is commonly called the family.

For example, a clinical trial might contain:

• One primary endpoint
• Three key secondary endpoints
• Several exploratory endpoints
• Multiple treatment comparisons
• Several dose comparisons
• Subgroup analyses

Whether these tests belong to the same multiplicity family depends on the scientific question and the prespecified testing strategy.

Notation for Multiple Testing

Suppose there are \(m\) hypotheses:

After testing them, each hypothesis is either rejected or not rejected. Let:

• \(V\) = number of rejected true null hypotheses
• \(S\) = number of rejected false null hypotheses
• \(R=V+S\) = total number of rejected hypotheses
• \(m_0\) = number of true null hypotheses
• \(m_1=m-m_0\) = number of false null hypotheses

These quantities form the foundation of FWER and FDR.

Family-Wise Error Rate

The family-wise error rate is defined as the probability of making at least one false rejection within the family.

In words: FWER is the probability that at least one rejected hypothesis is actually a true null hypothesis.

A procedure that controls FWER at 0.05 guarantees, under the assumptions required by that procedure, that:

This is a very strong form of error control.

Why FWER Is Attractive in Confirmatory Trials

Suppose a regulatory claim is based on several hypotheses. If even one false positive could result in an unsupported efficacy claim, controlling FWER is highly relevant.

For example, a confirmatory clinical trial might have multiple primary or key secondary hypotheses whose results will support formal claims.

In this setting, the question is often: "What is the probability that I make at least one false efficacy claim?"

FWER directly addresses this question.

False Discovery Rate

The false discovery rate takes a different perspective. It is the expected proportion of false discoveries among all discoveries.

with the convention that:

Thus FDR focuses on the quality of the collection of discoveries rather than on whether the collection contains even one false discovery.

A Simple Example

Suppose a study rejects 20 hypotheses. Among those 20 discoveries, suppose 2 are actually false discoveries. Then:

The observed false discovery proportion is:

or 10%.

FDR is the expectation of this quantity over repeated experiments:

Therefore, an FDR procedure targeting 10% attempts to ensure that the expected false discovery proportion is no greater than 10%.

FDP vs. FDR

These terms are closely related but should not be confused.

The Central Conceptual Difference

The easiest way to remember the distinction is:

This difference explains why FDR procedures can be substantially more powerful than FWER procedures when many hypotheses are tested.

FWER Is More Stringent

Controlling FWER at 5% requires:

FDR control at 5% requires:

These are not equivalent requirements.

A procedure can permit some false discoveries while maintaining a low expected false discovery proportion.

This usually provides greater power when the goal is to identify many potential signals rather than guarantee that the entire set of discoveries contains no false positives.

Bonferroni Control of FWER

The classic Bonferroni procedure is one of the simplest methods for controlling FWER.

Suppose there are \(m\) hypotheses and the desired family-wise error rate is \(\alpha\). Instead of testing every hypothesis at \(\alpha\), test each one at:

For example, if:

then each hypothesis is tested at:

This guarantees FWER control by the Bonferroni inequality.

Bonferroni Using Adjusted P-Values

Instead of changing the significance threshold, one can adjust each p-value:

A hypothesis is rejected when:

Why Bonferroni Can Be Conservative

Bonferroni treats every test as if it needs a share of the entire error budget. When there are many hypotheses, the individual threshold can become extremely small.

For example, with 100 hypotheses and a family-wise alpha of 0.05:

A real signal may therefore fail to reach significance even when the effect is scientifically meaningful.

This loss of power is one reason that FDR procedures are attractive in exploratory, high-dimensional settings.

Holm's Step-Down Procedure

Holm's procedure also controls FWER but is generally less conservative than the simple Bonferroni correction.

First, sort the p-values:

Compare the smallest p-value with:

The second smallest is compared with:

and so on. The \(i\)-th ordered p-value is compared with:

The procedure stops at the first failure.

Benjamini–Hochberg Control of FDR

The most widely known FDR procedure is the Benjamini–Hochberg (BH) procedure.

Again, sort the p-values:

For each ordered p-value, calculate the BH critical value:

where:

• \(i\) = rank of the p-value
• \(m\) = total number of hypotheses
• \(q\) = desired FDR level

Find the largest \(i\) satisfying:

Then reject all hypotheses with ranks:

Worked Benjamini–Hochberg Example

Suppose ten hypotheses produce the following ordered p-values:

The largest rank satisfying the BH condition is:

Therefore, the first five hypotheses are rejected.

Bonferroni vs. Benjamini–Hochberg

Using the same ten p-values, compare Bonferroni and BH.

Bonferroni requires:

Only the p-values 0.001 and 0.004 pass.

The BH procedure rejects the first five hypotheses.

The BH procedure is more permissive because it controls a different and generally less stringent error criterion.

FWER vs. FDR: Side-by-Side

A Clinical Trial Example

Suppose a randomized clinical trial evaluates a new treatment on five secondary endpoints. The p-values are:

Suppose the family-wise error rate must be controlled at 5%. Using Bonferroni:

Only the p-value 0.004 is significant.

This may be appropriate if the study intends to make formal confirmatory claims about these endpoints.

What Happens Under FDR Control?

Suppose instead that the five endpoints are exploratory and the investigator wants to identify promising signals for future research. Using BH at:

the ordered critical values are:

The first four p-values satisfy their corresponding thresholds:

• 0.004 ≤ 0.01
• 0.012 ≤ 0.02
• 0.021 ≤ 0.03
• 0.038 ≤ 0.04

Thus, BH would identify the first four as discoveries.

Why FDR Can Have Much Greater Power

Suppose a study tests 10,000 biomarkers. If Bonferroni FWER control is used at 5%, the individual threshold is:

That is an extremely stringent threshold.

Many genuine but moderate signals may not pass it.

FDR control instead allows the threshold to increase with the rank of the p-value. This can produce substantially more discoveries while maintaining control over the expected false discovery proportion.

FWER in Confirmatory Clinical Research

FWER control is particularly important when the hypotheses correspond to formal claims. Examples include:

• Multiple primary endpoints
• Key secondary endpoints
• Multiple dose comparisons
• Multiple treatment comparisons
• Multiplicity arising from interim testing strategies
• Confirmatory subgroup claims

The reason is straightforward: a single false positive can lead to an unsupported conclusion.

FDR in Exploratory Research

FDR is often attractive when researchers are screening large numbers of hypotheses. Examples include:

• Genomic studies
• Transcriptomic studies
• Proteomic studies
• High-throughput screening
• Biomarker discovery
• Large-scale imaging analyses
• Exploratory subgroup or feature screening

In these settings, rejecting no hypotheses may be less useful than identifying a reasonably reliable collection of candidates for further investigation.

FDR Does Not Mean "5% Chance of a False Positive"

A common misunderstanding is: "FDR = 5% means each significant result has a 5% probability of being false."

That is not generally what FDR means.

FDR controls:

across repeated experiments under the assumptions of the procedure.

It does not automatically mean:

for every individual hypothesis.

FWER Does Not Mean Every P-Value Has Error Rate 5%

Likewise, FWER is a family-level criterion. A procedure controlling FWER at 5% guarantees:

It does not mean that every individual hypothesis has exactly a 5% probability of being falsely rejected.

The Role of Dependence

Hypothesis tests are often correlated. For example, clinical endpoints may be correlated, and biomarkers measured on the same patients are rarely independent.

Dependence matters because the probability of observing multiple small p-values changes with the correlation structure.

Some procedures are valid under broad dependence structures, while others require independence or specific forms of positive dependence.

Bonferroni Under Dependence

One reason Bonferroni remains popular is its simplicity and broad validity. Using the union bound:

If the individual error allocations satisfy:

then FWER is controlled at level \(\alpha\).

No independence assumption is required for this basic Bonferroni argument.

Weighted Bonferroni

Not all hypotheses necessarily have equal importance. A weighted Bonferroni strategy can allocate different portions of the error budget to different hypotheses. Let weights \(w_i\) satisfy:

Then hypothesis \(i\) can be tested at:

This can be useful when some hypotheses are more important than others.

FDR Is Not Always the Right Choice

The higher power of FDR procedures does not automatically make them preferable.

FDR may be inappropriate when:

• A single false claim would be unacceptable.
• The hypotheses support regulatory claims.
• The analysis is confirmatory rather than exploratory.
• There is a strong need to protect the family from any false rejection.
• The estimand and multiplicity strategy require stronger error control.

FWER Is Not Always the Right Choice

Conversely, FWER can be unnecessarily restrictive when the scientific goal is large-scale discovery.

For example, screening 20,000 biomarkers with Bonferroni correction may result in very few discoveries, even when many genuine signals exist.

In such settings, FDR control may provide a better balance between false discoveries and discovery power.

Relationship Between FWER and FDR

There is an important mathematical relationship between the two criteria. Because:

where \(I(\cdot)\) is an indicator function, it follows that:

under the standard convention that \(V/R=0\) when \(R=0\).

Therefore, controlling FWER at level \(\alpha\) also controls FDR at level \(\alpha\).

A Useful Extreme Example

Suppose a procedure always produces exactly 100 discoveries and, on average, 5 are false. Then:

An FDR of 5% may therefore be acceptable. But if every one of those experiments contains at least one false discovery, then:

Thus, an FDR procedure can have excellent FDR control while offering little FWER protection.

Why the Distinction Matters for Clinical Trials

Consider a trial with ten exploratory biomarkers. If the goal is to identify candidate biomarkers for future validation, allowing a small proportion of false discoveries may be reasonable.

If the goal is instead to make a definitive regulatory claim that a biomarker predicts treatment benefit, stronger multiplicity control may be required.

The statistical procedure should therefore follow the purpose of the inference.

R Example: Bonferroni

R makes Bonferroni adjustment straightforward.

p <- c(
  0.004, 0.012, 0.021,
  0.038, 0.120
)

p.adjust(p, method = "bonferroni")

The adjusted p-values are obtained by multiplying each p-value by the number of tests, with values capped at 1.

R Example: Holm

p.adjust(
  p,
  method = "holm"
)

Holm generally provides greater power than simple Bonferroni while maintaining FWER control.

R Example: Benjamini–Hochberg

p.adjust(
  p,
  method = "BH"
)

The resulting adjusted p-values can be compared with the desired FDR level. For example:

p_adj <- p.adjust(
  p,
  method = "BH"
)

p_adj <= 0.05

Manual Benjamini–Hochberg Calculation in R

The BH procedure can also be implemented explicitly to understand the algorithm.

p <- c(
  0.004, 0.012, 0.021,
  0.038, 0.120
)

q <- 0.05

m <- length(p)

ord <- order(p)

p_sorted <- p[ord]

critical <- (1:m) / m * q

comparison <- p_sorted <= critical

data.frame(
  rank = 1:m,
  p_value = p_sorted,
  critical_value = critical,
  reject_condition = comparison
)

Adjusted P-Values Under BH

For a sorted collection of p-values, BH-adjusted p-values are obtained by working backward through the ordered values. Conceptually:

The resulting values are then mapped back to the original hypothesis order.

FWER vs. FDR in a Decision Framework

Common Multiple-Testing Mistakes

1. Running many tests at 0.05 without adjustment. The family-level false-positive probability can become much larger than 5%.
2. Calling Bonferroni "the" multiple-testing correction. Bonferroni is one FWER procedure among many.
3. Assuming FDR and FWER are interchangeable. They control different quantities.
4. Interpreting FDR as the probability an individual result is false. FDR is a family-level expected proportion.
5. Using FDR automatically because it has greater power. Higher power is not the relevant criterion if the scientific claim requires strong FWER control.
6. Defining the testing family after looking at the results. The family and multiplicity strategy should generally be prespecified for confirmatory inference.
7. Ignoring dependence assumptions. Different procedures have different theoretical requirements.
8. Adjusting only the p-values that "look related." The multiplicity strategy should correspond to the complete inferential family, not simply the tests that happened to produce interesting results.
9. Reporting unadjusted p-values as confirmatory evidence. If multiplicity adjustment is required, the adjusted inference should be reported clearly.

FWER Procedures Commonly Used in Practice

FDR Procedures Commonly Used in Practice

Strong vs. Weak FWER Control

Another important concept is the distinction between weak and strong control of FWER.

Weak control concerns the complete null configuration, where all hypotheses are true. Strong control requires control of FWER under every possible configuration of true and false null hypotheses.

In confirmatory clinical research, strong control is generally the relevant standard.

must hold regardless of which hypotheses are actually true.

Why Strong Control Matters

Suppose five hypotheses are tested but only two are actually true. A procedure with weak FWER control under the complete null does not necessarily guarantee adequate error control in this mixed configuration.

A strong FWER procedure is designed to protect against all such configurations.

FWER and Multiplicity in Clinical Development

Multiplicity can arise from many sources in clinical development:

• Multiple treatment arms
• Multiple doses
• Multiple endpoints
• Multiple time points
• Multiple populations
• Multiple estimands
• Interim analyses
• Subgroup analyses
• Multiple testing strategies

These sources should not necessarily all be placed into one giant multiplicity family. The appropriate structure depends on the scientific claims and the prespecified testing strategy.

Multiplicity Is More Than P-Value Adjustment

A sophisticated multiplicity strategy may involve:

• Hierarchical testing
• Gatekeeping
• Alpha allocation
• Graphical approaches
• Closed testing
• Fallback procedures
• Sequential testing
• Combination tests

The objective is not simply to make p-values larger. The objective is to construct a valid inferential system for the complete family of claims.

Family-Wise Error Rate and Alpha Spending

Interim analyses create another form of multiplicity. If a hypothesis is tested repeatedly during a clinical trial, repeatedly using the same nominal significance threshold can inflate the overall type I error.

Sequential methods therefore allocate the total type I error across analyses.

Conceptually:

This is related to the broader principle of controlling error across a family of statistical opportunities.

FWER vs. FDR Is a Scientific Decision

The choice should not be made solely because one procedure produces more significant results.

Ask what happens after the statistical analysis.

If a significant result will support a definitive claim, stronger error control may be appropriate.

If a significant result will instead identify a candidate for further study, FDR may provide a more useful balance between discovery and false positives.

A Compact Numerical Comparison

Suppose 100 hypotheses are tested at an overall target of 5%.

The unadjusted approach has the greatest nominal power but can have severe false-positive inflation. Bonferroni has strong FWER control but may be conservative. Holm retains FWER control while generally improving power. BH controls FDR and can identify substantially more discoveries in large families.

What Should Be Reported?

A multiple-testing analysis should clearly report:

• The hypotheses included in the testing family
• The primary error criterion
• The nominal target level
• The adjustment procedure
• Whether the procedure was prespecified
• The adjusted p-values or adjusted decision thresholds
• The assumptions relevant to the procedure
• How multiplicity interacts with the interpretation of the results

Example Statistical Analysis Plan Language

For a confirmatory family, the analysis plan might specify that the family-wise type I error rate will be controlled at the 5% level using a prespecified step-down procedure.

For an exploratory biomarker analysis, the plan might instead specify that false discovery rate will be controlled at 5% using the Benjamini–Hochberg procedure.

The critical point is that the chosen framework should be documented before interpreting the results.

A Practical Decision Table

The Most Important Mathematical Difference

The two criteria can be remembered through two equations. FWER:

FDR:

FWER cares about whether the number of false discoveries is zero or greater than zero. FDR cares about the fraction of discoveries that are false.

A Useful Mental Model

Common Interview Question

A common biostatistics interview question is: "What is the difference between FWER and FDR?"

A strong answer is:

Another Common Interview Question

"Does controlling FWER at 5% also control FDR at 5%?"

Yes. Since:

taking expectations gives:

Therefore, a procedure that strongly controls FWER at 5% also controls FDR at 5%. The reverse implication does not generally hold.

Another Common Interview Question

"Why is Benjamini–Hochberg more powerful than Bonferroni?"

The key is that they control different quantities. Bonferroni protects against the probability of any false rejection. BH controls the expected proportion of false discoveries. The latter permits some false discoveries and therefore generally allows a larger rejection set.

Summary of the Major Procedures

Practical Workflow

Final Takeaways

The distinction between FWER and FDR is fundamental to modern statistical inference.

FWER controls the probability of making at least one false discovery:

FDR controls the expected proportion of false discoveries among the discoveries:

FWER is generally more conservative and is particularly important for confirmatory claims where a single false positive can be consequential.

FDR is generally more powerful and is particularly useful when the objective is to discover a collection of promising signals among a very large number of hypotheses.

References

Bonferroni, C.E. (1936). Teoria statistica delle classi e calcolo delle probabilità. Pubblicazioni del R. Istituto Superiore di Scienze Economiche e Commerciali di Firenze.
Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2), 65–70.
Benjamini, Y. & Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B, 57(1), 289–300.
Benjamini, Y. & Yekutieli, D. (2001). The control of the false discovery rate under dependency. Annals of Statistics, 29(4), 1165–1188.
Hochberg, Y. (1988). A sharper Bonferroni procedure for multiple tests of significance. Biometrika, 75(4), 800–802.
Hommel, G. (1988). A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika, 75(2), 383–386.
Westfall, P.H. & Young, S.S. (1993). Resampling-Based Multiple Testing: Examples and Methods for P-Value Adjustment. Wiley.