Why the Hypothesis Family Matters
A genomic analysis may test many associations for several phenotypes, several annotations, or combinations of both. Before interpreting a discovery list, specify which hypotheses the list represents. Applying a false discovery rate (FDR) procedure separately to each phenotype is not the same analysis as applying one procedure to all phenotype–variant tests together. The resulting discoveries, error-rate guarantees, and scientific claims can differ.
A hypothesis family is the collection of hypotheses to which a multiple-testing procedure is applied as one set. For example, the tests of all variants for one pre-specified phenotype could form a family. Alternatively, all tests across all phenotypes could form one pooled family. A family may also be defined by annotation, such as tests for variants in a pre-specified functional category. The family definition should match the intended scope of the claim: “discoveries for this phenotype” and “discoveries across all phenotypes examined” are different scopes.
For the Benjamini–Hochberg (BH) procedure, sort the \(m\) p-values in the specified family as \(p_{(1)}\le\cdots\le p_{(m)}\). At target level \(q\), find the largest \(k\) such that \(p_{(k)}\le kq/m\), and reject hypotheses corresponding to \(p_{(1)},\ldots,p_{(k)}\). The family size \(m\) is part of the calculation. Under independent tests, or suitable positive dependence such as PRDS with valid marginal p-values, BH controls FDR at level \(q\). Under other dependence structures that do not meet those conditions, this guarantee should not be presumed; the dependence material from earlier in the course covers that qualification.
Separate Families and a Pooled Family
In a separate-family analysis, each phenotype (or annotation) has its own list and its own multiplicity adjustment. This can be appropriate when the scientific questions are genuinely distinct and the analysis plan specifies those families in advance. The resulting statement is about FDR within each family, under the procedure’s assumptions. It is not automatically a statement that the combined list across families has FDR at most \(q\).
In a pooled analysis, all hypotheses in the stated scope form one family, and the procedure uses the total number of tests. This directly targets the FDR among the pooled set of rejections. A pooled analysis may be preferable when the scientific claim is a single discovery list spanning phenotypes or annotations, or when the analysis seeks to control errors across all reported claims.
Worked Example: Separate Phenotype Families Versus Pooling
Suppose two phenotype analyses are pre-specified. Phenotype A has p-values \(0.01\) and \(0.04\). Phenotype B has 18 p-values, all at least \(0.20\). Apply BH at \(q=0.05\), first separately and then to all 20 tests pooled.
Separate analysis. For A, \(m=2\), so the BH thresholds are \(0.05/2=0.025\) and \(2(0.05)/2=0.05\). The second sorted p-value, \(0.04\), is at most \(0.05\); both A hypotheses are rejected. For B, every p-value is at least \(0.20\), whereas every BH threshold is at most \(0.05\), so no B hypotheses are rejected.
Pooled analysis. Now \(m=20\). The first two sorted p-values are \(0.01\) and \(0.04\), and the first two thresholds are \(0.05/20=0.0025\) and \(2(0.05)/20=0.005\). Neither p-value meets its corresponding threshold. The remaining p-values are at least \(0.20\), while all thresholds are at most \(0.05\). Thus no pooled hypotheses are rejected.
The difference is not an arithmetic inconsistency: separate testing uses a family size of two for phenotype A, while pooling uses a family size of 20. The separate result supports a phenotype-A-family claim under its stated assumptions. It does not provide the same error-rate target as a single pooled discovery list.
What Separate FDR Guarantees Do—and Do Not—Combine Into
If the families are indexed by \(j\), write \(R_j\) and \(V_j\) for their rejection and false-rejection counts. When there is at least one total rejection, the pooled FDP can be written as
This is a weighted average of the family-specific FDPs, with weights determined by the realized rejection counts. Taking expectations does not generally turn it into a fixed weighted average of the family-specific FDRs: the weights are random and can be related to the FDPs. Therefore, the fact that each family was analyzed at level \(q\) does not, by itself, establish pooled FDR control at \(q\).
One safe way to target pooled FDR is to define the pooled family in advance and apply an appropriate procedure to that complete set of tests, under its assumptions. There are also procedures designed for structured families or selective claims, but their guarantees depend on their specific rules and assumptions. Simply running BH within each group and then collecting every rejection is not a substitute for such a guarantee.
Worked Example: A Pooled List Can Change the Rejections
Suppose three variants are tested for each of two phenotypes, giving p-values \(0.001, 0.012, 0.030\) for phenotype A and \(0.020, 0.080, 0.400\) for phenotype B. Compare separate BH analyses with one pooled BH analysis at \(q=0.05\).
Worked Example: Comparing Separate and Pooled BH Results
Separate analyses. Each family has \(m=3\), so its ordered thresholds are \(0.0167,\ 0.0333,\ 0.0500\).
For phenotype A, the largest p-value is \(0.030\), and \(0.030\le 0.0500\). Thus BH rejects all three hypotheses in A. For phenotype B, the ordered p-values are \(0.020,\ 0.080,\ 0.400\). The first fails its threshold because \(0.020>0.0167\); the second and third also fail because \(0.080>0.0333\) and \(0.400>0.0500\). BH makes no rejections in B.
Pooled analysis. There are \(m=6\) tests. In increasing order, the p-values are \(0.001,\ 0.012,\ 0.020,\ 0.030,\ 0.080,\ 0.400\). The thresholds are approximately \(0.00833,\ 0.0167,\ 0.0250,\ 0.0333,\ 0.0417,\ 0.0500\). At rank 4, \(0.030\le 0.0333\), while at rank 5, \(0.080>0.0417\); rank 4 is the largest passing rank. The pooled procedure therefore rejects the first four hypotheses.
Here pooling adds the phenotype-B p-value \(0.020\) to the rejection set while leaving the two larger B p-values unrejected. The separate and pooled analyses answer different questions and use different family sizes. The result does not imply that either procedure is universally more powerful: the outcome depends on the p-values and family definition.
Data-Dependent Family Selection
A family is data-dependent when the observed results help determine which tests, phenotypes, or annotations will be included in the reported analysis. For example, an analyst might inspect results for many phenotypes, choose the phenotype with the smallest p-value, and then run or report a nominally adjusted analysis only for that phenotype. The selected family is no longer equivalent to one fixed in advance. The search that led to the selection is part of the inferential process.
The issue can be demonstrated even with a single test in each candidate family. Suppose two phenotypes each have one true null hypothesis, and their p-values are independent and Uniform\((0,1)\). Select the phenotype with the smaller p-value and report a rejection if that selected p-value is at most \(0.05\).
Worked Example: Selecting the Smaller of Two Null P-Values
Let the two p-values be \(P_1\) and \(P_2\). The selected value is \(\min(P_1,P_2)\). A rejection occurs if either p-value is at most \(0.05\). By independence,
Under the global null, every rejection is false, so the FDP equals 1 when there is a rejection and 0 otherwise. The FDR is therefore \(0.0975\), not \(0.05\). Treating the selected phenotype as if it had been the only candidate family ignores the selection step. A procedure applied to the full, pre-specified set of hypotheses—or an appropriate method that accounts for selection—is needed for a guarantee covering this search.
Selection does not automatically make every analysis invalid. A family may be chosen using external information that is independent of the testing data, or a procedure may explicitly account for selection. But when selection uses the same results being tested, nominal FDR interpretations require justification; the fact that the final list is small or biologically plausible is not that justification.
Annotations, Overlap, and the Unit of a Discovery
Annotations introduce a further question: what exactly counts as one hypothesis? If a variant belongs to two annotations, its association test may appear in both annotation-specific lists. Those can be distinct scientific claims—for example, “this variant is associated in annotation A” and “this variant is associated in annotation B”—or they may be repeated representations of one underlying variant–phenotype claim. The family and the discovery count should reflect the intended claim.
If overlapping annotation lists are pooled by stacking their rows, a repeated test may be counted more than once. The resulting FDR is then defined over the stacked hypothesis list, not necessarily over unique variant–phenotype associations. Conversely, deduplicating tests changes the family and the interpretation. Neither choice is universally correct; the analysis should state whether hypotheses are annotation-specific or deduplicated, how membership was defined, and how a discovery appearing in multiple annotations is reported.
Annotation membership fixed by an external reference can define families before testing. If annotations are created, filtered, or selected using the same association results, the family selection itself may be data-dependent. For instance, choosing annotations because they contain unusually small p-values and then reporting within-annotation FDR as though those annotations were pre-specified fails to account for the search. The relevant object is the full selection-and-testing rule, not only the final list.
Practical Checks and Common Pitfalls
Before interpreting results across phenotypes or annotations, record the family definition, the number of tests in each family, whether the families overlap, and whether any family was selected after inspecting the data. Also identify the exact error-rate claim: per-family FDR or pooled FDR. Verify that the p-values are valid for the tests and that the dependence assumptions required by the chosen procedure are credible for the stated guarantee.
A useful comparison is to report both separate and pooled results when both scopes matter, while clearly labeling their distinct interpretations. Separate analyses can answer phenotype-specific questions; a pooled analysis can address the combined set of claims. If the scientific goal is to identify a phenotype first and then make claims within it, use a method whose guarantee accounts for that selection rather than silently treating the selected phenotype as pre-specified.
- Pitfall: “Every phenotype was analyzed at FDR 5%, so the combined list has FDR 5%.” Correction: Separate guarantees do not generally establish pooled FDR control; define and control the pooled family if that is the claim.
- Pitfall: “We selected the phenotype with the strongest result, then adjusted only its tests.” Correction: The selection used the testing results, so the analysis must account for the search or use independent selection information.
- Pitfall: “A variant reported in two annotations is automatically two independent discoveries.” Correction: Specify whether the claims are annotation-specific or whether the repeated test represents one unique association.
Decide whether discoveries are to be interpreted within phenotypes, within annotations, or across the full analysis.
List the hypotheses included, note overlaps, and specify whether repeated tests correspond to distinct claims.
Determine whether the family was specified independently of the testing results or selected using them.
Use per-family procedures for per-family claims, or a suitable procedure for the pooled or selection-adjusted claim intended.
Check Your Understanding
Answer each question, then compare with the solutions below.
- Two phenotypes are analyzed separately with BH at \(q=0.05\). What can be claimed from those per-phenotype guarantees, and what cannot be concluded about the combined list?
- For one fixed family, \(m=4\) and the sorted p-values are \(0.003,\ 0.018,\ 0.031,\ 0.210\). At \(q=0.05\), how many hypotheses does BH reject?
- Two independent true-null p-values are examined, and the smaller is reported as significant if it is at most \(0.05\). What is the probability of a rejection?
- A variant association appears in two overlapping annotation lists. What must be specified before interpreting an FDR calculated from the combined lists?
Solutions. (1) The guarantees concern FDR within each specified phenotype family; they do not, by themselves, establish that the combined list has FDR at most \(0.05\). (2) The thresholds are \(0.0125,\ 0.0250,\ 0.0375,\ 0.0500\). Rank 3 passes because \(0.031\le0.0375\), while rank 4 fails because \(0.210>0.0500\); BH rejects the first three. (3) The probability is \(1-(0.95)^2=0.0975\). (4) Specify whether the two entries are distinct annotation-specific hypotheses or one repeated variant–phenotype claim, and define the family and discovery count accordingly.