Interpreting FDR As An Expected Error Rate

From One Analysis to Repeated Analyses

A multiple-testing procedure produces a set of rejections from one observed dataset. In genomic applications, those rejections may be reported as candidate genes, variants, or other findings. The false discovery rate (FDR) is often described as an error rate, but its definition is about the procedure’s behavior across repeated analyses—not a promise that a particular result set contains a specified fraction of false findings.

To interpret an FDR target correctly, keep two quantities separate. The false discovery proportion describes the rejections in one analysis. The false discovery rate is the expected value of that proportion under repeated sampling, with the procedure and the hypothesis family specified. A realized proportion can be greater than the expected-rate target without contradicting FDR control.

Definition: For a specified family of hypotheses, let $R$ be the number rejected and $V$ the number of rejected hypotheses that are in fact true null hypotheses. The false discovery proportion is $$ \mathrm{FDP}=\frac{V}{R\vee 1}, $$ where $R\vee 1=\max(R,1)$. Thus $\mathrm{FDP}=0$ when $R=0$. The false discovery rate is $$ \mathrm{FDR}=E(\mathrm{FDP}), $$ where the expectation is over repeated data generated under the assumed model, for a fixed procedure and hypothesis family.

In a single analysis, $R$ is observable, but $V$ usually is not: the true null status of each rejected genomic hypothesis is unknown. In a hypothetical setting where the truth labels are known, the realized FDP can be calculated. In an actual study, the FDR is a property of the procedure and data-generating setup; it generally cannot be read off as the exact false fraction among the reported findings.

The expectation is taken over repeated datasets under the stated sampling model. In the usual frequentist interpretation, the set of true and false null hypotheses is treated as fixed while the data vary. If the procedure has been designed to control FDR at level $q$, the claim is typically $\mathrm{FDR}\leq q$ under its required conditions. It is not a claim that the FDP in every dataset is at most $q$, nor that the FDR equals $q$.

What an FDR Target Says

For the Benjamini–Hochberg (BH) procedure, the preceding material establishes the step-up rule and its control result under independence. In particular, for a family of $m$ tests with mutually independent p-values and valid, super-uniform p-values for true null hypotheses, BH at target $q$ controls the FDR at no more than $q$. The target is a bound on the expectation of the FDP over repeated analyses under these conditions.

For example, an FDR target of $q=0.05$ does not say that no more than five percent of the rejections in this dataset are false. It does not say that each reported finding has a five percent probability of being false. It says that, over repeated datasets under the stated conditions, the expected value of $V/(R\vee1)$ is at most 0.05. The distinction matters even when the same procedure and the same target are used consistently.

Worked Example: A Realized FDP Above the Target

Suppose a hypothetical genomic analysis tests $m=10{,}000$ hypotheses using BH at $q=0.05$. In this particular analysis, it rejects $R=40$ hypotheses. For illustration, suppose the true statuses were known and $V=8$ of those rejections were true nulls. Calculate the realized FDP.

Substitute the observed counts into the definition:

\mathrm{FDP}=\frac{V}{R\vee1} =\frac{8}{40} =0.20.

The realized FDP is 0.20, or 20% of the 40 rejections. This result does not by itself contradict an FDR bound of 0.05. FDR control concerns the expected FDP across repeated datasets, not a maximum FDP imposed on each individual dataset. This example also assumes knowledge of the true statuses, which is ordinarily unavailable in a real genomic analysis.

Why FDR Is Not a Ratio of Expected Counts

The random denominator $R$ is essential. FDR is the expectation of a ratio, $E\{V/(R\vee1)\}$, not generally the ratio $E(V)/E(R)$. These are different summaries because the number of rejections varies from one dataset to another and may vary alongside the number of false rejections.

Justification: Let $I_i$ equal 1 if hypothesis $i$ is rejected and 0 otherwise, and let $H_i$ equal 1 if hypothesis $i$ is a true null and 0 otherwise. Then $R=\sum_{i=1}^{m} I_i$, $V=\sum_{i=1}^{m}H_iI_i$, and $$ \mathrm{FDR} =E\left(\frac{\sum_{i=1}^{m}H_iI_i}{R\vee1}\right) =\sum_{i=1}^{m}H_i E\left(\frac{I_i}{R\vee1}\right). $$ The second equality follows by linearity of expectation, with the fixed truth indicator $H_i$ taken outside the expectation. It shows that each possible true-null rejection contributes in relation to the total number rejected in that same analysis. Replacing this expectation of a ratio with a ratio of expectations changes the quantity being calculated.

Worked Example: The Ratio of Expectations Can Mislead

Consider a hypothetical procedure applied to the same fixed family in two equally likely repeated analyses. In the first analysis it makes $R=1$ rejection, which is false, so $V=1$. In the second it makes $R=99$ rejections and none is false, so $V=0$. Calculate both the FDR and the ratio $E(V)/E(R)$.

The FDPs in the two analyses are $1/1=1$ and $0/99=0$. Because the analyses are equally likely,

\mathrm{FDR}=E(\mathrm{FDP}) =\frac{1}{2}(1)+\frac{1}{2}(0) =0.50.

The expected false count is $E(V)=\{1+0\}/2=0.5$, while the expected rejection count is $E(R)=\{1+99\}/2=50$. Therefore,

\frac{E(V)}{E(R)} =\frac{0.5}{50} =0.01.

The two values, 0.50 and 0.01, differ substantially. The ratio $E(V)/E(R)$ gives more influence to the analysis with many rejections; the FDR averages the FDP of each repeated analysis. This hypothetical calculation illustrates the definitions and is not an example of a procedure controlling FDR at a particular target.

Procedure-Level Control Is Not a Per-Finding Probability

An FDR target also does not assign a probability of being false to each rejected hypothesis. The target concerns the expected fraction of false rejections in the full rejection set. It does not identify which findings are false, and it does not imply that each finding has the same chance of being false. A probability statement about a particular hypothesis would require a different inferential quantity and assumptions appropriate to that question.

Worked Example: Interpreting a Reported Rejection Set

Suppose a hypothetical study reports $R=80$ rejections from a procedure configured for FDR control at $q=0.10$. A reader says, “Each of these 80 findings has a 10% probability of being false.” Is that interpretation justified?

No. The target $q=0.10$ is a bound on the expected FDP across repeated analyses, subject to the procedure’s assumptions. It is not a probability attached separately to each of the 80 findings. Even if the realized false proportion in this analysis were known to be $8/80=0.10$, that would describe the set as a whole, not establish a 10% false probability for every member. In an actual study, the true $V$ is generally unknown, so even that realized proportion cannot usually be calculated.

Assumptions, Diagnostics, and Limits

An interpretation is only as sound as the family and conditions to which it refers. The FDR statement applies to the specified hypothesis family and the procedure used for that family. If tests are omitted, added, or split into families after looking at results, the resulting interpretation may no longer match the procedure’s stated target. Define the family before analysis and report it clearly.

For the BH control result under independence, valid null p-values and mutual independence are relevant assumptions. A procedure’s numerical target alone does not establish those conditions. In genomic data, dependence among tests can occur; the independence result should not be presented as a universal guarantee. The dependence-specific procedures and conditions require separate consideration.

A useful diagnostic is to distinguish what can be checked from what cannot. Analysts can document the family, method, target, and rejection count, and can assess whether the analysis followed its prespecified plan. They generally cannot verify the exact FDP from one real dataset because the true null status of each rejected hypothesis is unknown. Nor can one dataset empirically establish the repeated-sampling expectation. Simulation can examine performance under an explicitly specified model, but its conclusions depend on that model.

FDR control is often less stringent than familywise error rate control, which focuses on the probability of making at least one false rejection. These procedures address different error summaries; neither target should be substituted for the other when interpreting results. An FDR target is useful when the goal is to manage the average false fraction among discoveries, but it does not guarantee a small false fraction in every individual result set.

Common Mistakes

Wrong: “BH at 5% means at most 5% of today’s findings are false.” Correction: The target bounds the expected FDP over repeated datasets under the required conditions; one realized FDP can exceed 5%.
Wrong: “Every selected gene has a 5% probability of being false.” Correction: FDR is a set-level expected error proportion, not a per-finding probability.
Wrong: “The FDR is $E(V)/E(R)$.” Correction: It is $E\{V/(R\vee1)\}$; averaging the per-analysis proportions is not the same as taking a ratio of average counts.
Wrong: “The target equals the observed false fraction.” Correction: The target is a chosen control level, while the observed FDP is a realized quantity and usually cannot be known exactly in genomic data.

Key takeaway: FDP describes one analysis; FDR is its expectation across repeated analyses. An FDR target is a procedure-level expected-error bound under stated conditions—not a guarantee for every dataset and not a probability that any particular discovery is false.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A procedure rejects 25 hypotheses, of which 3 are true nulls. What is the realized FDP?
In a repeated-analysis framework, what does an FDR target of 0.05 constrain, and what does it not guarantee?
Why is $E\{V/(R\vee1)\}$ generally not equal to $E(V)/E(R)$?
If no hypotheses are rejected, what is the FDP under the stated convention?

Solutions. (1) $\mathrm{FDP}=3/25=0.12$, or 12% for that analysis. (2) It bounds the expected FDP across repeated analyses under the procedure’s conditions; it does not guarantee that every realized FDP is at most 0.05 or assign a 5% probability of being false to each finding. (3) The random denominator can vary with the numerator, so the expectation of the per-analysis ratio need not equal a ratio of expectations. (4) If $R=0$, then $R\vee1=1$ and $V=0$, so $\mathrm{FDP}=0/1=0$.