Why the Family Is Part of the Claim
A multiple-testing procedure controls an error rate for a specified collection of hypotheses. It does not, by itself, determine which scientific questions belong together. That decision matters: a result can have valid familywise error rate (FWER) control for one prespecified collection and still fail to support a broader claim that readers might reasonably take from the study.
In genomic analysis, the possible testing opportunities may include genes, variants, phenotypes, time points, subgroups, or contrasts between treatment groups. The family should therefore be set in relation to the study’s aims and planned claims, not chosen only after the analyst sees which p-values are small. There is no universal rule that all tests in a research program must form one family, or that every endpoint must be combined into one family. The key is to define the scope clearly and make the error guarantee match the scope of the conclusions.
The family is best understood as part of the inferential plan. Ask: which hypotheses will be examined, which results will be used to make a particular claim, and across what set of claims is a false positive meant to be limited? A family may correspond, for example, to all gene-level tests for one primary phenotype. If the study also tests several phenotypes and will make a broad claim based on whichever produces a finding, the relevant scope is wider than the gene-level tests for one phenotype alone.
What Makes a Family Definition Defensible?
A defensible definition is tied to scientific aims and decisions, specified before the results are inspected, and stated transparently. It should identify the hypotheses included and explain why the resulting claims are appropriately limited. It is not enough to label a set “primary” or “exploratory” if the actual analysis and reporting let results from outside that set influence the stated conclusion.
Separate families can be reasonable when they correspond to genuinely distinct, prespecified questions and the conclusions are reported separately. For instance, a primary phenotype and a secondary phenotype might have distinct scientific roles, with a separate error guarantee for each. But two nominally separate families do not together have the same guarantee as one family covering both. If the study claims a discovery across either phenotype, the reader needs an error statement that covers the opportunity to make that combined claim.
The central distinction is between scope of control and scope of interpretation. The former is the hypotheses included in the procedure; the latter is the set of conclusions readers are asked to accept. If the interpretation is broader, more selective, or more opportunistic than the controlled family, the stated error guarantee does not justify it.
Worked Example: A Prespecified Gene Set for One Primary Phenotype
Suppose a hypothetical genomic study names one primary phenotype and, before examining results, specifies four gene-level null hypotheses for that phenotype. The investigators apply a Bonferroni procedure at \(\alpha=0.05\). Assume each p-value is valid under its null; Bonferroni does not require independence among these tests.
For \(m=4\) tests, the per-test threshold is $$ \frac{\alpha}{m}=\frac{0.05}{4}=0.0125. $$ Suppose the four p-values are \(0.004\), \(0.011\), \(0.018\), and \(0.040\). Comparing each with \(0.0125\), the first two null hypotheses are rejected and the last two are not.
The procedure controls the probability of one or more false rejections among these four tests at no more than \(0.05\), under the stated validity assumption. It does not provide the same guarantee for additional genes, additional phenotypes, or other comparisons that were examined but omitted from the defined family. The conclusion must stay at the scope supported by the four-test family.
A Short Justification for Scope-Specific Control
The union bound explains why the number and identity of tests in the family matter. If each of \(m\) valid null p-values is tested at \(\alpha/m\), the probability of at least one false rejection is bounded by the sum of the individual false-rejection probabilities. This argument does not require the tests to be independent.
Here \(I_0\) is the set of true null hypotheses in the family, so \(|I_0|\leq m\). The conclusion is about that family because the union in the first line is over its true nulls. Adding tests adds possible false-rejection events; leaving relevant tests out means the bound does not cover them. The same scope principle applies when another valid FWER procedure is used, even though its detailed assumptions and calculation may differ.
Worked Example: Separate Families and a Broader Claim
Consider a hypothetical study with two prespecified phenotypes. For each phenotype, investigators test two gene hypotheses as a separate family and use Bonferroni control at \(0.05\) within each family. The p-values for phenotype A are \(0.012\) and \(0.030\); those for phenotype B are \(0.018\) and \(0.041\).
Within each two-test family, the threshold is $$ \frac{0.05}{2}=0.025. $$ Thus, the \(0.012\) test in family A and the \(0.018\) test in family B pass their respective thresholds; \(0.030\) and \(0.041\) do not. The analysis supports two separate within-family FWER statements, one for each phenotype.
Now suppose the intended headline is instead “a gene association was found for at least one of the two phenotypes.” This combines the opportunities across both families. The two separate guarantees do not imply a study-wide FWER of \(0.05\) for that combined claim. As a simple bound, the probability of at least one false rejection across the two families is no greater than \(0.05+0.05=0.10\); that bound does not assert the actual error rate is \(0.10\).
If the four hypotheses were treated as one family, the Bonferroni threshold would be \(0.05/4=0.0125\). Only the p-value \(0.012\) passes. This comparison shows why an analysis plan must match the intended claim: separate family-level conclusions and a single claim across either phenotype have different scopes.
Unplanned Contrasts and Selective Family Definitions
A frequent scope error occurs when the analyst examines results, chooses the most promising contrast, and then treats that contrast as though it had been the only planned test. The choice itself used information from multiple opportunities. If the reported claim is based on the winner, the family—or a suitable selection-aware procedure—must account for the selection process.
The relevant universe is not necessarily every comparison that could ever be imagined. It is the set of comparisons that were actually considered, or that the analysis plan allowed to influence the claim. If six genotype contrasts were all inspected and the smallest p-value was highlighted, reporting an unadjusted single-test threshold ignores the five other chances to obtain a small p-value. A prespecified, limited set of contrasts may be treated differently from an open-ended search, but the set and its role should be documented.
Worked Example: Correcting for Choosing the Smallest P-Value
In a hypothetical study, investigators inspect six planned genotype contrasts for a phenotype and then select the contrast with the smallest p-value for emphasis. Suppose the six p-values are \(0.008\), \(0.014\), \(0.021\), \(0.033\), \(0.047\), and \(0.080\). Assume all six p-values are valid under their respective null hypotheses.
If the six contrasts form the family, Bonferroni’s threshold at \(\alpha=0.05\) is $$ \frac{0.05}{6}=0.008333\ldots. $$ Only \(0.008\) is below this threshold. Its Bonferroni-adjusted p-value is $$ \min(1,6\times 0.008)=0.048. $$ Thus, under this family definition, the selected contrast just passes the \(0.05\) FWER criterion.
If the analyst instead reports \(0.008\) as an unadjusted p-value and compares it with \(0.05\), the arithmetic for a single test is correct but the inferential scope is not: the contrast was chosen from six inspected opportunities. The family-level conclusion is limited to the six listed contrasts, and the Bonferroni guarantee relies on valid p-values for those tests. If further outcomes or contrasts also informed selection, they would need to be reflected in the scope of the analysis or handled by an appropriate method.
Diagnostics, Reporting, and Alternatives
A practical diagnostic is to reconstruct the path from study aim to reported claim. Write down the primary scientific question, enumerate the hypotheses that could support its answer, and compare that list with the set actually covered by the correction. Then check whether any analysis choices—such as selecting an endpoint, subgroup, model, or contrast—were made after results were seen. If they were, determine whether the selection was included in the family or addressed by a valid selection-aware design or analysis.
Specify the scientific conclusion the analysis is intended to support, including which phenotype, population, and type of contrast it concerns.
Identify the hypotheses, endpoints, contrasts, and other choices whose results could have led to that claim.
Check that the stated FWER procedure covers those opportunities, and that its assumptions, including p-value validity, are appropriate.
Name the family and state whether the error guarantee is per family or covers a broader set of conclusions.
The main alternatives are not merely different formulas. One option is a single family covering all hypotheses relevant to a broad claim. This gives a direct guarantee for that claim but can be more conservative than controlling error separately for narrower questions. Another is to define distinct families for prespecified, scientifically distinct aims and report each family’s guarantee separately. A third is to use a design or method that explicitly accounts for selection or for a planned sequence of decisions. Whatever approach is chosen, the guarantee must be stated at the scope it actually covers.
A family need not be selected only by a rigid “all tests together” rule. Separate families can be defensible when they are genuinely prespecified and conclusions remain distinct. Conversely, labeling analyses as separate does not make their results independent claims if the paper emphasizes whichever is most favorable. Independence between tests, where present, also does not by itself solve a scope mismatch: separate within-family guarantees remain separate guarantees.
A useful report names the hypotheses in the family, the procedure and target \(\alpha\), any prespecified exclusions, and any data-dependent selection. It should distinguish confirmatory conclusions covered by the plan from exploratory findings that suggest hypotheses for further study. Exploratory results can be scientifically useful, but a confirmatory FWER statement should not be attached to a set defined after inspecting the same results unless the selection is properly accounted for.
Common Scope Errors
- Defining the family after seeing the p-values. Keeping only favorable tests and correcting within that reduced set does not generally control the selection process. Define the family prospectively or use an analysis that accounts for selection.
- Reporting only the selected contrast. A smallest p-value among several inspected contrasts is not automatically a single-test result. Include the relevant contrast opportunities in the inferential scope.
- Combining separate guarantees into a larger one. Control at \(0.05\) within each of two families does not establish control at \(0.05\) across their union. State the separate scope or use a procedure for the broader claim.
- Confusing a list of tests with the set of claims. A narrow correction may be mathematically valid for the tests listed but still fail to support a broader headline. Ensure the interpretation does not outrun the family.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A study tests eight genes for one prespecified primary phenotype using Bonferroni at \(0.05\). What is the threshold, and which p-values, \(0.006\) and \(0.009\), pass?
- Two distinct, prespecified phenotype families each have FWER controlled at \(0.05\). May the investigators claim that FWER is \(0.05\) across a conclusion based on a discovery in either family? Explain.
- An analyst examines five contrasts, reports the smallest p-value \(0.012\), and calls it significant because it is below \(0.05\). What is the scope error? What is the Bonferroni threshold for the five contrasts?
- Why can a family containing only tests that were significant after analysis undermine a confirmatory FWER claim?
Solutions. (1) The threshold is \(0.05/8=0.00625\). The p-value \(0.006\) passes; \(0.009\) does not. (2) No. Each guarantee applies within its own family, not automatically to the combined “either family” claim. (3) The analyst ignored the selection from five opportunities. The Bonferroni threshold is \(0.05/5=0.01\), so \(0.012\) does not pass. (4) The family was defined using the results being tested, excluding unsuccessful opportunities and conditioning the scope on observed outcomes. The resulting within-set calculation does not generally account for the selection process; a prespecified family or a valid selection-aware analysis is needed.