Why the Testing Family Must Be Defined
A GWAS can produce an enormous number of association results. Before interpreting a threshold for “genome-wide” evidence, it is necessary to say exactly which hypotheses were tested and which claims are being treated as part of one inferential family. The same numerical threshold can refer to very different scopes: one phenotype across the genome, several phenotypes across the genome, or a limited set of variants at a specified locus.
A testing family is not determined simply by the number of rows in a results file. It is the set of hypotheses to which a stated error-control claim applies. Defining that set requires specifying the scientific claims being considered, the units of testing, and the analyses included. This tutorial focuses on that definition; it does not prescribe a particular multiple-testing procedure.
What Is One GWAS Hypothesis?
For example, let \(G_{ij}\) denote the dosage of an effect allele for participant \(i\) at variant \(j\), and let \(Y_{ik}\) be phenotype \(k\). A simple quantitative-trait association model might be
where \(\mathbf{C}_i\) contains prespecified covariates. The corresponding null hypothesis for that variant and phenotype is \(H_{0,jk}:\beta_{jk}=0\); the alternative is that the dosage coefficient is not zero. This is a hypothesis about the specified model and coefficient, not a general statement that the variant has no possible biological effect. A case-control analysis may instead specify a regression model and test its variant coefficient on the log-odds scale.
The model, phenotype definition, variant representation, covariates, and tested parameter help identify the hypothesis. Changing one of these may create a distinct test. For example, an additive dosage test and a genotype-category test are not automatically the same hypothesis. Nor are the association tests for two different phenotypes, even if those phenotypes are correlated.
A test is commonly indexed by a variant and a phenotype, but the family can include more dimensions. If the analysis tests several genetic models, ancestry groups, phenotype definitions, or interaction terms, those tests may all contribute hypotheses to the family when any of them can support a reported claim. Which dimensions belong together is a design and inferential decision, not a fact determined by the data format.
Worked Example: Counting a Variant-by-Phenotype Family
Suppose a hypothetical study prespecifies one additive association test for each of 720,000 variants and each of three quantitative phenotypes. It uses the same covariate-adjusted linear-regression specification for all tests. How many hypotheses are in the joint family?
There is one hypothesis for each variant-phenotype pair. Thus,
The factor of 1 represents the single prespecified model per pair. The joint family therefore contains 2,160,000 hypotheses. If the study instead defines one family per phenotype, each such family has 720,000 hypotheses. These are different family definitions: a claim about the joint family concerns tests across all three phenotypes, whereas a claim about a phenotype-specific family concerns tests for only that phenotype.
This count assumes that each variant passes the prespecified analysis criteria and contributes one test for each phenotype. If some tests are omitted because of missingness or a prespecified quality filter, the actual membership and count must reflect those rules. The number itself does not establish that any particular error-control method is valid.
Genome-Wide, Phenotype-Specific, and Locus-Specific Families
A genome-wide family for one phenotype includes the specified association hypotheses for that phenotype across the analyzed variants. A joint multi-phenotype family includes hypotheses across several phenotypes, usually across the analyzed variants as well. A locus-specific family includes hypotheses for a defined genomic region, such as a prespecified interval, rather than for the whole set of analyzed variants.
These definitions answer different questions. If the scientific claim is that at least one variant anywhere in the genome is associated with a specified phenotype, the relevant family should include the genome-wide search for that phenotype. If the claim may be made for any of several phenotypes, a joint family can reflect that broader opportunity for a discovery. If a study has a separately specified question about one locus, a locus-specific family may be appropriate for that question—but it does not, by itself, cover a search across the rest of the genome.
A locus must also be defined clearly enough to identify its tests. It might be a prespecified interval or a region defined by a stated rule. A locus-specific family that includes every tested variant in that region is different from a single hypothesis about a selected lead variant. Selecting a lead variant after looking across many variants does not turn the original search into a one-test analysis.
Worked Example: A Locus Family Is Not a Genome-Wide Family
Imagine a hypothetical study that analyzes 500,000 variants for one phenotype and has a separate, prespecified follow-up question about a region containing 240 of those variants. For the genome-wide question, the family contains 500,000 variant hypotheses. For the specified region-only question, the family contains 240 hypotheses.
Suppose the reported association is the smallest \(p\)-value among those 240 regional tests. It is a result selected from a 240-hypothesis search, not a result from a single variant chosen before the data were examined. An error-control statement for the regional family would address false rejections among those 240 tests; it would not automatically address the original genome-wide search.
Conversely, if the region was selected because it looked most promising after examining all 500,000 tests, describing only the 240 variants in that region as the family would omit the selection process. The narrow family would not represent the full set of opportunities that led to the reported claim. The appropriate scope depends on the search and selection that actually occurred, not merely on the final region highlighted in a report.
Families Depend on the Claims and Analyses Considered
A useful way to define a family is to list the hypotheses that could each lead to a claim of discovery within the stated analysis. Consider what was prespecified and what was examined before selecting results. If investigators tested multiple phenotype definitions, several genetic models, or separate ancestry strata and could report any successful result as evidence for an association, those tests may need to be represented in the family for that claim.
Not every analysis in a research program must be placed in one enormous family. Distinct scientific questions can have distinct families when they are genuinely specified and interpreted separately. But labeling analyses as separate does not itself make them separate in a meaningful inferential sense. If the choice of which analysis to report depends on which one produced the most striking result, the broader set of examined analyses matters to the scope of the claim.
A discovery analysis and a genuinely distinct, prespecified replication analysis may have different inferential roles. The family for a discovery claim is not automatically the family for the replication claim. The report should state which tests belong to each stage and what conclusion each stage is intended to support. Likewise, a family restricted to a primary phenotype does not cover secondary-phenotype claims unless those tests are included in the stated scope.
Worked Example: Multiple Models and Phenotypes
A hypothetical analysis tests 300,000 variants for two phenotypes. For each variant-phenotype pair, investigators run an additive model and a dominant model, and they are willing to report a discovery from either model. Assume both models are successfully fit for every pair and each gives one hypothesis.
For one phenotype, the count is
hypotheses, because there are two model-specific tests per variant. If the family covers both phenotypes and both models, its count is
The first factor is the number of variants, the second is the number of phenotypes, and the third is the number of models. If the analysis had instead designated one model as primary in advance and treated the other as exploratory, the primary family could be narrower, but the exploratory result should not be presented as though it had been covered by the primary family’s error-control statement.
What Error Control Means for a Defined Family
The family determines the meaning of an error-rate statement. To make that precise, let the family be \(H_1,\ldots,H_m\). A false rejection occurs when a procedure rejects a null hypothesis that is true. Let \(V\) be the number of true null hypotheses rejected. The family-wise error rate (FWER) is the probability of at least one such false rejection in the family:
This definition makes the role of family membership explicit: changing which hypotheses are in the family changes the event \(V\geq 1\). A statement that FWER is controlled at level \(\alpha\) concerns the probability of one or more false rejections within the specified family, under the conditions required by the chosen procedure. It does not mean that the probability a particular reported association is false equals \(\alpha\), nor that the family definition alone provides control.
A Short Justification: Why Scope Changes the Error Event
Suppose a family contains two sets of hypotheses, \(A\) and \(B\), and let \(E_A\) be the event of at least one false rejection in \(A\), with \(E_B\) defined similarly for \(B\). For the combined family, a false rejection occurs if either event occurs. Therefore,
The combined-family error probability is consequently
which is an event defined over both sets, not just over \(A\). No independence assumption is needed for this identity. The probability may depend on how the tests relate, but the logical event already shows why a claim restricted to \(A\) does not describe the error event for \(A\cup B\). The same reasoning applies when \(A\) is one phenotype and \(B\) is another, or when one set is a primary model and the other a secondary model.
Some procedures aim to control error under the global null, when every hypothesis in the family is true; others seek control even when some hypotheses are false. These are different guarantees. In either case, the family must be stated to know which hypotheses and which false-rejection event the guarantee concerns. The family definition and the choice of procedure are related but separate decisions.
Dependence Does Not Merge Distinct Hypotheses
Nearby variants can be correlated because of linkage disequilibrium, and phenotypes can also be correlated. This dependence affects the joint behavior of association statistics and may matter for the choice or calibration of a multiple-testing procedure. It does not, by itself, make two variant-phenotype hypotheses identical. If two variants are separately tested and either can be reported, both are members of the family unless the analysis explicitly defines a different testing unit.
Similarly, one should not infer that the number of independent tests equals the number of hypotheses. A family is a collection of claims, not a count of independent random variables. Procedures can account for dependence in different ways; defining the family does not require pretending that its tests are independent.
Common Mistake: Defining the Family After Seeing the Results
- Wrong reasoning: “Only one phenotype and one lead variant appear in the paper, so the family contains one test.”
- Correction: Identify the full search that could have produced the reported claim. If the lead variant was selected from many variants, or the phenotype from several examined phenotypes, the relevant family may include those opportunities.
- Wrong reasoning: “The variants are correlated, so they count as one hypothesis.”
- Correction: Correlation among test statistics does not erase distinct hypotheses. State the tested units and let the chosen procedure address dependence as appropriate.
- Wrong reasoning: “A genome-wide error-control statement automatically covers every phenotype and model in the project.”
- Correction: It covers only the family to which it applies. State whether the family includes one phenotype or several, one model or several, and any additional analyses that could support a reported discovery.
Check Your Understanding
Use the stated search and claim—not just the final reported result—to answer these questions.
- A study tests 400,000 variants for one phenotype under one prespecified model. How many hypotheses are in its variant-wide family?
- A study tests the same 400,000 variants for four phenotypes and may report a discovery for any phenotype. What is the size of the joint variant-by-phenotype family if there is one model per pair?
- Why does selecting the smallest \(p\)-value among 80 variants in a region not make the analysis a single-hypothesis test?
- Two nearby variants have correlated association statistics. Does that mean they are automatically one hypothesis? Explain.
- In your own words, what event does an FWER statement describe for a specified family?