False Discovery Rate (FDR) Control

Introduction

Modern statistical analyses frequently involve testing many hypotheses at the same time.

For example, a clinical trial may evaluate treatment effects across multiple biomarkers, laboratory parameters, exploratory endpoints, patient subgroups, or molecular measurements.

Similarly, genomic and transcriptomic analyses can involve thousands or even millions of simultaneous statistical tests.

If every hypothesis is tested at the ordinary 5% significance level, a large number of apparently significant findings can occur simply because many tests are being performed.

This is the multiple-testing problem.

One response is to control the probability of making any false positive, using procedures such as Bonferroni or Holm.

Another strategy is to control the expected proportion of rejected hypotheses that are false discoveries.

This quantity is the false discovery rate, or FDR.

The Multiple-Testing Problem

Suppose one hypothesis is tested at significance level:

If the null hypothesis is true, the probability of obtaining a false-positive result is at most 5%.

Now suppose 20 independent true null hypotheses are tested, each at 5%.

The probability of obtaining at least one false-positive result is:

which is approximately:

or approximately 64.2%.

Thus, when many tests are performed, using 0.05 independently for every test can produce a substantial probability of at least one false positive.

Family-Wise Error Rate vs. False Discovery Rate

Two important error concepts are the:

• Family-wise error rate (FWER)
• False discovery rate (FDR)

The family-wise error rate is the probability of making at least one false rejection within the family of hypotheses.

where \(V\) is the number of false rejections.

FDR instead concerns the proportion of rejected hypotheses that are false.

with the convention that the ratio is defined as zero when \(R=0\).

Here:

• \(V\) = number of false discoveries
• \(R\) = total number of rejected hypotheses

A Simple Example of the Difference

Suppose an analysis identifies 100 statistically significant findings.

Imagine that approximately five of these findings are false discoveries.

The proportion of false discoveries is:

or 5%.

An FDR-controlling procedure at 5% is designed to control this type of expected proportion.

A FWER-controlling procedure has a much stricter objective: it attempts to keep the probability of even one false rejection below the specified level.

The Four Fundamental Quantities

The multiple-testing framework can be represented using four quantities.

Let:

where:

• \(m\) = total number of hypotheses tested
• \(m_0\) = number of true null hypotheses
• \(m_1\) = number of false null hypotheses

Among the rejected hypotheses:

where:

• \(V\) = false rejections
• \(S\) = true rejections

Defining the False Discovery Rate

The false discovery proportion for one particular experiment is:

when \(R>0\).

The false discovery rate is the expected value of this quantity:

Thus, FDR is an expectation over repeated applications of the statistical procedure.

The Benjamini–Hochberg Procedure

The most widely known method for controlling FDR is the Benjamini–Hochberg (BH) procedure.

Suppose there are \(m\) p-values:

First, sort them from smallest to largest:

The ordered p-values are then compared with the BH critical values:

where \(q\) is the desired FDR level.

More conventionally, the \(i\)-th ordered p-value is compared with:

The BH Decision Rule

Find the largest index \(k\) satisfying:

Then reject all hypotheses corresponding to:

If no such \(k\) exists, no hypotheses are rejected.

Why the Largest Valid \(k\) Matters

The BH procedure does not simply ask whether each p-value is below its own individual threshold.

Instead, it identifies the largest index \(k\) satisfying the criterion.

Once \(k\) is found, all hypotheses with ordered p-values through \(p_{(k)}\) are rejected.

This creates a step-up procedure.

A Complete Worked Example

Suppose an exploratory analysis evaluates eight hypotheses.

The unadjusted p-values are:

We want to control the FDR at:

There are:

hypotheses.

Step 1: Sort the P-Values

Arrange the p-values from smallest to largest.

Step 2: Calculate the BH Thresholds

The BH threshold for rank \(i\) is:

The resulting thresholds are:

Step 3: Find the Largest Valid Rank

The largest rank satisfying:

is:

Therefore, the BH procedure rejects the first four ordered hypotheses.

Step 4: Identify the Rejected Hypotheses

The first four ordered p-values correspond to:

• H3, \(p=0.001\)
• H6, \(p=0.008\)
• H1, \(p=0.013\)
• H8, \(p=0.021\)

Therefore:

The remaining four hypotheses are not rejected by the BH procedure.

Why H4 Is Not Rejected

Notice that H4 has an unadjusted p-value of only 0.040.

Under an ordinary unadjusted 5% threshold:

so it would be called statistically significant.

Under BH control, however, H4 corresponds to rank 5, whose threshold is:

Since:

H4 is not rejected.

The Step-Up Nature of BH

The BH procedure is called a step-up procedure because it starts with the smallest p-values and searches upward for the largest rank that satisfies the criterion.

This differs from Holm's step-down procedure.

FDR Control Is Less Conservative Than FWER Control

One of the main reasons to use FDR control is that it generally permits more discoveries than procedures designed to control FWER at the same nominal level.

Consider \(m=100\) hypotheses.

A Bonferroni procedure at 5% uses:

Every p-value must therefore be at or below 0.0005 to be rejected.

The BH procedure uses rank-specific thresholds:

For example:

Thus, the BH procedure becomes progressively less stringent as the rank increases.

Why This Makes Sense

Suppose many very small p-values are observed.

The presence of a large cluster of small p-values provides evidence that the analysis contains genuine signals rather than only random noise.

The BH procedure takes this information into account.

A collection of small p-values can therefore support a larger rejection set while maintaining control of the expected false discovery proportion under the procedure's assumptions.

FDR Level vs. Individual P-Value Threshold

A common misunderstanding is to treat:

as though it means every rejected hypothesis has a 5% probability of being false.

That is not what FDR control means.

The 5% level concerns the expected proportion of false discoveries among the rejected hypotheses.

FDR and the Number of Discoveries

Suppose the BH procedure produces:

discoveries.

An FDR target of 5% does not imply that exactly one of those 20 findings is false.

The realized number of false discoveries can vary from analysis to analysis.

FDR is a long-run expectation:

The guarantee is about the statistical procedure across repeated experiments, not a direct probability statement about each individual finding.

What Happens When No Hypotheses Are Rejected?

If no ordered p-value satisfies:

then there is no valid \(k\).

The procedure rejects nothing.

This is completely valid.

FDR control does not require the analysis to identify discoveries.

FDR When All Null Hypotheses Are True

Consider the special case in which every hypothesis is actually null.

Then every rejection is a false discovery.

Therefore, whenever \(R>0\):

and hence:

If \(R=0\), the false discovery proportion is defined as zero.

Therefore:

This relationship is useful for understanding the distinction between FDR and FWER.

When every hypothesis is null, controlling FDR is equivalent to controlling the probability of making any rejection.

FDR and FWER Are Not Interchangeable

Although both concepts address false positives, they answer different questions.

Bonferroni vs. Benjamini–Hochberg

The difference can be summarized simply.

Bonferroni controls:

BH controls:

under the conditions required by the BH procedure.

Thus, Bonferroni is concerned with avoiding any false discovery, while BH allows some false discoveries provided their expected proportion among discoveries is controlled.

Holm vs. Benjamini–Hochberg

Holm's procedure is a step-down method for controlling FWER.

BH is a step-up method for controlling FDR.

When FDR Control Is Attractive

FDR control is particularly useful when an analysis involves a large number of related hypotheses and the goal is to identify promising signals.

Examples include:

• Genomic analyses
• Transcriptomic analyses
• Proteomic analyses
• High-dimensional biomarker screening
• Exploratory subgroup investigations
• Large panels of laboratory measurements
• Imaging feature analyses
• High-throughput screening
• Exploratory secondary or tertiary endpoints

FDR in Clinical Trials

FDR control can have a role in clinical research, but it should not automatically replace traditional multiplicity procedures.

For example, a clinical development program may distinguish between:

• Confirmatory primary endpoints
• Key secondary endpoints
• Exploratory biomarkers
• Exploratory subgroup analyses
• Hypothesis-generating safety signals

A confirmatory endpoint may require strong FWER control because a false-positive claim could support an incorrect regulatory or clinical conclusion.

An exploratory biomarker analysis may instead prioritize identifying a useful set of candidate signals for subsequent confirmation.

FDR Does Not Automatically Solve Multiplicity

It is important not to interpret FDR control as a universal replacement for all multiplicity procedures.

The correct procedure depends on the scientific objective.

For example, if a trial has one primary endpoint and several key secondary endpoints whose statistical significance will support confirmatory claims, FWER control may be more appropriate.

If the analysis instead examines hundreds of exploratory biomarkers and the goal is to identify a manageable collection of candidates for further study, FDR control may be more useful.

The Benjamini–Hochberg Algorithm Step by Step

Benjamini–Hochberg Adjusted P-Values

Instead of reporting only the reject/non-reject decision at a particular FDR level, analysts can calculate BH-adjusted p-values.

For ordered p-values, the raw quantity is:

However, these values may not be monotonic.

Therefore, the adjusted p-values are calculated using a reverse cumulative minimum:

The adjusted values are also capped at 1:

These adjusted p-values are then mapped back to the original hypotheses.

Adjusted P-Values for the Worked Example

Recall the ordered p-values:

Thus, at FDR level 5%, the hypotheses with BH-adjusted p-values below 0.05 are the first four.

The fifth adjusted p-value is:

so H4 is not rejected.

BH Adjusted P-Values Are Not the Same as Raw P-Values

An adjusted p-value should be interpreted in the context of the selected multiple-testing procedure.

For example, an adjusted p-value of 0.042 means that the hypothesis would be rejected by the BH procedure at an FDR threshold of 5%.

It does not mean that the posterior probability that the null hypothesis is true is 4.2%.

R Implementation

R provides a direct implementation of the Benjamini–Hochberg procedure through the p.adjust() function.

p_values <- c(
  0.013,
  0.650,
  0.001,
  0.040,
  0.120,
  0.008,
  0.300,
  0.021
)

p_adjusted <- p.adjust(
  p_values,
  method = "BH"
)

p_adjusted

The returned vector contains the BH-adjusted p-values in the same order as the original hypotheses.

Identify Significant Findings at FDR = 5%

alpha_fdr <- 0.05

significant <- p_adjusted <= alpha_fdr

significant

The resulting logical vector identifies the hypotheses rejected under the BH procedure.

Compare Multiple-Testing Procedures in R

R can calculate several commonly used multiplicity adjustments.

bonferroni <- p.adjust(
  p_values,
  method = "bonferroni"
)

holm <- p.adjust(
  p_values,
  method = "holm"
)

hochberg <- p.adjust(
  p_values,
  method = "hochberg"
)

bh <- p.adjust(
  p_values,
  method = "BH"
)

results <- data.frame(
  raw_p = p_values,
  bonferroni = bonferroni,
  holm = holm,
  hochberg = hochberg,
  BH = bh
)

results

This is useful for understanding how the choice of error criterion affects the resulting discoveries.

BH vs. Benjamini–Yekutieli

The standard BH procedure has its strongest classical guarantee under independence and under certain forms of positive dependence among the test statistics.

When arbitrary dependence among test statistics is a concern, the Benjamini–Yekutieli (BY) procedure provides a more general FDR-control approach.

The BY procedure modifies the BH threshold using the harmonic sum:

The BY critical value becomes:

Because:

the BY procedure is more conservative than ordinary BH.

BH vs. BY

Dependence Between Tests

Multiple tests are often correlated.

For example, laboratory measurements may be biologically related, gene expression measurements can exhibit correlation, and multiple endpoints in a clinical trial may be correlated.

Therefore, the dependence structure among tests should be considered when selecting an FDR procedure.

What Does "Controlling FDR at 5%" Actually Mean?

Suppose a procedure controls FDR at:

The intended statement is:

under the relevant assumptions.

This does not mean:

• Exactly 5% of the findings will be false.
• Each finding has a 5% probability of being false.
• The probability that a specific null hypothesis is true is 5%.
• The false discovery proportion in the current dataset cannot exceed 5%.

The realized false discovery proportion can be larger than 5% in an individual dataset.

FDR and q-Values

The term q-value is often used in discussions of FDR, but it requires careful interpretation.

In the BH context, adjusted p-values can be used to determine which findings would be rejected at a specified FDR threshold.

The term q-value is also used more broadly for the minimum estimated FDR at which a particular observation would be called significant, especially in methods based on estimating the proportion of true null hypotheses.

Estimating the Proportion of True Null Hypotheses

The basic BH procedure does not require an explicit estimate of the proportion of true null hypotheses.

Let:

where \(\pi_0\) is the proportion of hypotheses that are truly null.

The standard BH procedure effectively uses a conservative approach rather than estimating \(\pi_0\) from the observed p-values.

Other FDR methods can estimate \(\pi_0\), potentially increasing power when many hypotheses are truly non-null.

FDR and Exploratory Biomarker Analysis

Consider a clinical study evaluating 2,000 biomarkers.

Testing every biomarker at an unadjusted 5% level could result in a large number of nominally significant findings even if most biomarkers have no association with treatment response.

A researcher may instead specify:

and apply BH control.

The resulting discovery set is then designed to have controlled expected false-discovery proportion at the specified level under the procedure's assumptions.

This can be considerably more useful for discovery than requiring FWER control at 5% across all 2,000 biomarkers.

FDR and Secondary Endpoints

Secondary endpoints require particular care in clinical trials.

Not every collection of secondary endpoints should automatically be treated as an FDR family.

Investigators should first define the inferential objectives.

For example:

• Which endpoints support confirmatory claims?
• Which endpoints are exploratory?
• Are the hypotheses logically related?
• Is a specific ordering of testing required?
• Is strong control of false-positive claims required?
• Will regulatory or labeling decisions depend on the results?

These questions determine whether FDR, FWER, hierarchical testing, gatekeeping, or another multiplicity strategy is appropriate.

Defining the Testing Family

A critical practical issue is deciding which hypotheses belong to the multiplicity family.

For example, suppose a trial tests:

• One primary endpoint
• Five key secondary endpoints
• 100 exploratory biomarkers

It would generally be inappropriate simply to throw all 106 tests into one undifferentiated FDR analysis without considering the scientific structure of the trial.

The testing family should be defined based on the objectives of the analysis and the claims that will be made.

FDR Does Not Mean "Ignore Multiple Testing"

A common misconception is:

"If the analysis is exploratory, there is no need to adjust the p-values."

That is not correct.

Exploratory analyses can still involve extensive multiple testing.

FDR provides one principled way of accounting for that multiplicity while retaining more discovery power than strict FWER procedures.

A Larger Worked Example

Suppose a biomarker program evaluates 15 biomarkers.

The sorted p-values are:

Let:

The BH threshold at rank \(i\) is:

The critical values are:

The largest valid rank is:

Therefore, the first seven hypotheses are rejected.

Why You Do Not Stop at the First Failure

This is one of the most common implementation mistakes.

Suppose the comparison looks like:

The rank-3 failure does not mean the procedure automatically stops there.

Rank 4 passes.

Therefore:

and all four hypotheses are rejected.

The BH procedure specifically asks for the largest valid rank.

Common Mistakes With BH

1. Comparing every p-value only with 0.05. That ignores the multiple-testing procedure.
2. Using \(q/m\) for every hypothesis. BH uses rank-specific thresholds \(iq/m\).
3. Stopping at the first failed comparison. The largest valid rank must be identified.
4. Forgetting to sort the p-values. The BH algorithm operates on ordered p-values.
5. Calling \(q\) the probability that an individual result is false. FDR is an expected proportion across the rejection set.
6. Assuming FDR control means no false positives. It explicitly permits some false discoveries.
7. Using FDR automatically for confirmatory claims. The appropriate multiplicity objective depends on the scientific and regulatory purpose.
8. Ignoring dependence among tests. Dependence can affect the assumptions and operating characteristics.
9. Defining the testing family after seeing the results. The multiplicity family should generally be prospectively defined.
10. Confusing BH-adjusted p-values with Storey q-values. They are related but can be calculated using different methodologies.

FDR vs. No Multiplicity Adjustment

Choosing the FDR Level

Common choices include:

or:

The appropriate level depends on the scientific purpose.

A highly exploratory screening analysis may reasonably use a less stringent FDR target than a setting in which false discoveries have substantial consequences.

The level should be chosen before examining the results that determine the final discovery set.

FDR and Statistical Power

Multiplicity adjustment generally reduces the probability of declaring a finding significant compared with completely unadjusted testing.

However, FDR procedures are generally less conservative than FWER procedures.

Consequently, FDR control can provide substantially greater power to identify true signals in high-dimensional settings.

This is particularly important when:

• Many hypotheses are being tested.
• A meaningful fraction of hypotheses may truly be non-null.
• The objective is discovery rather than definitive confirmation.
• A moderate proportion of false discoveries can be tolerated.

FDR and Replication

FDR control does not eliminate the need for replication.

A biomarker identified using FDR-adjusted analysis may represent a promising candidate rather than a definitive biological finding.

Replication in an independent dataset can provide important evidence that the finding is robust.

FDR in Omics Research

FDR control became particularly influential in high-dimensional biological research because thousands of hypotheses may be tested simultaneously.

For example, an RNA-sequencing analysis may evaluate differential expression for thousands of genes.

If each gene is tested at \(p<0.05\) without multiplicity adjustment, hundreds of nominally significant findings can arise simply from the large number of tests.

Applying BH control creates a discovery set with a controlled expected false-discovery proportion under the method's assumptions.

FDR and Exploratory Safety Analyses

Large safety databases can also involve many statistical comparisons.

However, safety analyses require particular caution.

A rare but clinically important adverse event should not necessarily be dismissed simply because it does not survive a multiplicity adjustment.

Conversely, a statistically significant signal after FDR adjustment does not automatically establish causality.

Statistical multiplicity control is only one component of signal assessment.

FDR Does Not Replace Clinical Judgment

A p-value, adjusted or otherwise, does not determine whether a finding is clinically meaningful.

A complete interpretation should consider:

• Effect size
• Confidence interval
• Clinical relevance
• Biological plausibility
• Multiplicity
• Study design
• Data quality
• Potential biases
• Replication

Reporting FDR-Adjusted Results

A statistical report should clearly state:

• The family of hypotheses tested
• The number of hypotheses in the family
• The multiplicity procedure used
• The target FDR level
• Whether BH or another FDR procedure was used
• The dependence assumptions or rationale
• The number of discoveries
• The adjusted p-values
• The corresponding effect estimates and confidence intervals where appropriate

For example:

FDR and Confidence Intervals

A common reporting issue is that adjusted p-values may be presented without effect estimates.

Where possible, results should also provide:

• Estimated treatment effect
• Confidence interval
• Raw p-value
• Multiplicity-adjusted p-value

This allows readers to distinguish statistical evidence from the magnitude and precision of the effect.

A Useful Results Table

This is generally more informative than presenting only a list of raw p-values.

Manual BH Calculation in R

The BH procedure can also be implemented directly rather than using p.adjust().

p_values <- c(
  0.013,
  0.650,
  0.001,
  0.040,
  0.120,
  0.008,
  0.300,
  0.021
)

m <- length(p_values)
q <- 0.05

ord <- order(p_values)

p_sorted <- p_values[ord]

rank <- seq_along(p_sorted)

threshold <- rank * q / m

comparison <- data.frame(
  rank = rank,
  p_value = p_sorted,
  threshold = threshold,
  pass = p_sorted <= threshold
)

comparison

Find the Largest Valid Rank in R

valid <- which(
  p_sorted <= threshold
)

if (length(valid) == 0) {

  k <- 0

} else {

  k <- max(valid)
}

k

For the worked example:

k
# 4

Therefore, the first four ordered hypotheses are rejected.

Create the BH Rejection Indicator

reject_sorted <- rep(FALSE, m)

if (k > 0) {
  reject_sorted[1:k] <- TRUE
}

reject <- rep(FALSE, m)

reject[ord] <- reject_sorted

data.frame(
  raw_p = p_values,
  reject = reject
)

The final vector is returned to the original hypothesis order.

Calculate BH Adjusted P-Values Manually

adjusted_sorted <- p_sorted * m / rank

adjusted_sorted <- rev(
  cummin(rev(adjusted_sorted))
)

adjusted_sorted <- pmin(
  adjusted_sorted,
  1
)

adjusted <- numeric(m)

adjusted[ord] <- adjusted_sorted

data.frame(
  raw_p = p_values,
  BH_adjusted_p = adjusted
)

This reproduces the basic calculation used by the BH adjustment.

A Compact Function for BH Testing

bh_test <- function(p_values, q = 0.05) {

  m <- length(p_values)

  ord <- order(p_values)

  p_sorted <- p_values[ord]

  rank <- seq_len(m)

  threshold <- rank * q / m

  valid <- which(
    p_sorted <= threshold
  )

  reject_sorted <- rep(FALSE, m)

  if (length(valid) > 0) {

    k <- max(valid)

    reject_sorted[seq_len(k)] <- TRUE

  } else {

    k <- 0

  }

  reject <- rep(FALSE, m)

  reject[ord] <- reject_sorted

  list(
    k = k,
    reject = reject,
    ordered_p = p_sorted,
    threshold = threshold
  )
}

The function can then be used as:

result <- bh_test(
  p_values,
  q = 0.05
)

result$k
result$reject

How BH Behaves as the FDR Level Changes

The selected FDR threshold directly affects the rejection set.

For example:

p.adjust(
  p_values,
  method = "BH"
)

p.adjust(
  p_values,
  method = "BH"
) <= 0.01

p.adjust(
  p_values,
  method = "BH"
) <= 0.05

p.adjust(
  p_values,
  method = "BH"
) <= 0.10

A larger FDR target generally permits a larger set of discoveries.

FDR at 1%, 5%, and 10%

The appropriate choice depends on the consequences of false discoveries and the purpose of the analysis.

FDR and Replication Studies

Suppose an exploratory study evaluates 10,000 biomarkers and identifies 300 discoveries after FDR control.

Those 300 findings may be carried forward into an independent validation study.

The second study may use a different multiplicity strategy because its scientific purpose is now confirmatory rather than exploratory.

This illustrates a useful development strategy:

FDR and Hierarchical Testing

Some clinical studies have structured hypotheses in which one test should only be interpreted after another test succeeds.

For example:

• Primary endpoint
• Key secondary endpoint
• Additional secondary endpoint
• Exploratory biomarker endpoint

A hierarchical or gatekeeping strategy may therefore be more appropriate than simply applying BH to all p-values simultaneously.

FDR and Multiplicity Families

The phrase "control FDR at 5%" is incomplete unless the relevant family of hypotheses is clear.

For example:

could refer to:

• All biomarkers
• All genes
• All secondary endpoints
• All subgroup comparisons
• One prespecified subset of hypotheses

These are different statistical questions.

Pre-Specification Is Important

The following should generally be specified before the analysis:

• Testing family
• FDR target
• Multiplicity procedure
• Primary analysis population
• Handling of missing data
• Rules for defining the p-values
• Any dependence considerations
• Rules for reporting discoveries

Changing the family or FDR threshold after seeing the results can undermine the intended error-control properties.

FDR Does Not Control the Maximum Number of False Discoveries

Suppose 100 hypotheses are rejected under a procedure controlling FDR at 5%.

It is incorrect to state:

"At most five of these findings can be false."

That is not the FDR guarantee.

The realized number of false discoveries can exceed five.

The guarantee concerns the expected proportion:

FDR vs. False Discovery Proportion

These two terms are closely related but not identical.

This distinction is fundamental.

FDR and the "Probability a Discovery Is False"

In informal scientific discussion, researchers sometimes say: "With FDR controlled at 5%, only 5% of significant findings are expected to be false."

This can be a useful shorthand, but it should not be interpreted as a literal posterior probability for each individual finding.

The precise statistical statement is:

The distinction becomes particularly important when communicating statistical results to non-statistical audiences.

When Bonferroni May Be Preferable

A FWER-controlling method may be preferable when:

• A single false-positive claim would have serious consequences.
• The analysis is confirmatory.
• Regulatory claims depend on statistical significance.
• The number of hypotheses is relatively small.
• The hypotheses are key clinical claims.
• Strong control of false-positive probability is required.

When FDR May Be Preferable

FDR control may be attractive when:

• The analysis contains many hypotheses.
• The primary goal is discovery.
• Some false discoveries can be tolerated.
• Identifying a broad set of candidate signals is valuable.
• FWER control would be excessively conservative.
• Results will be subjected to independent validation.

Practical Decision Framework

Summary of the Benjamini–Hochberg Procedure

The Core Formula

If you remember only one formula from this tutorial, remember:

where \(k\) is the largest rank satisfying the inequality.

Worked Example at a Glance

Common Interview Question: What Does FDR Control?

A strong answer is:

Common Interview Question: How Does BH Work?

A concise answer is:

Common Interview Question: Why Is FDR Less Conservative?

FWER control attempts to prevent even one false discovery with high probability.

FDR control instead permits some false discoveries as long as their expected proportion among discoveries is controlled.

This less stringent objective can provide substantially greater power when many hypotheses are tested.

Common Interview Question: Is BH Always Valid Under Dependence?

The classical BH guarantee is available under independence and extends to certain positive dependence structures.

For more general arbitrary dependence, the Benjamini–Yekutieli procedure provides a more conservative FDR-control approach.

Therefore, the dependence structure should be considered when selecting an FDR procedure.

Common Interview Question: Does FDR = 5% Mean 5% of My Findings Are False?

Not literally for one dataset.

The formal guarantee concerns the expected false discovery proportion:

The realized proportion in a particular dataset can be higher or lower.

Statistical Analysis Plan Considerations

If FDR control is part of an analysis plan, the document should clearly describe:

• The hypothesis family
• The number of hypotheses
• The target FDR level
• The selected FDR procedure
• The assumptions concerning dependence
• The method for calculating adjusted p-values
• The significance criterion
• The treatment of missing or unevaluable measurements
• The approach to multiplicity across different endpoint families
• The distinction between exploratory and confirmatory conclusions

A Practical Reporting Workflow

FDR vs. FWER: The Big Picture

The Most Important Concept

The most important distinction is this:

versus:

FWER asks whether any false discovery occurs.

FDR asks how large the false-discovery fraction is expected to be among the discoveries.

That difference explains why FDR procedures can identify substantially more signals in high-dimensional analyses while still providing formal statistical error control.

References

Benjamini, Y. & Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B, 57(1), 289–300.
Benjamini, Y. & Yekutieli, D. (2001). The control of the false discovery rate in multiple testing under dependency. Annals of Statistics, 29(4), 1165–1188.
Storey, J.D. (2002). A direct approach to false discovery rates. Journal of the Royal Statistical Society: Series B, 64(3), 479–498.
Hochberg, Y. & Benjamini, Y. (1990). More powerful procedures for multiple significance testing. Statistics in Medicine, 9, 811–818.
Shaffer, J.P. (1995). Multiple hypothesis testing. Annual Review of Psychology, 46, 561–584.