Why Multiplicity Matters in G×E Studies
A gene–environment (G×E) study may test many variants, exposures, genotype codings, and interaction models. Even when every null hypothesis is true, a large collection of tests can produce small p-values by chance. A p-value of 0.05 is calibrated for one test under its null; it does not mean that a study running hundreds of tests has only a 5% chance of any false positive.
Multiplicity is not merely a bookkeeping problem. A study may report the smallest p-value after trying several exposure definitions or interaction scales, even if those alternatives were not all listed in the initial analysis plan. The relevant correction depends on which hypotheses could have led to the reported claim. A defensible analysis therefore defines its testing family, distinguishes planned tests from searches, and specifies how discovery findings will be evaluated in independent data.
The family should correspond to the scientific claims, not simply to whichever set of results is convenient after analysis. If a study searched across 200 variants, three exposures, and two interaction scales before highlighting a result, the effective search involved those alternatives. Counting only the tests in the final table would understate the search. Conversely, tests addressing genuinely separate, prespecified questions may be assigned to distinct families if the analysis plan explains that choice and the claims are kept distinct.
Controlling the Chance of Any False Positive
The Bonferroni procedure controls FWER without requiring independence among tests. Suppose \(m\) valid p-values \(p_1,\ldots,p_m\) are tested and the desired family-wise level is \(\alpha\). Reject a null hypothesis only if its p-value is at most \(\alpha/m\). Equivalently, multiply each p-value by \(m\), truncating the result at 1, to obtain a Bonferroni-adjusted p-value. The procedure can be conservative, especially when tests are correlated, as can occur for related genetic variants or overlapping exposures.
Holm’s step-down procedure also controls FWER under arbitrary dependence and is never less powerful than the simple Bonferroni procedure. Sort the p-values from smallest to largest, \(p_{(1)}\leq\cdots\leq p_{(m)}\). Compare \(p_{(1)}\) with \(\alpha/m\). If it passes, compare \(p_{(2)}\) with \(\alpha/(m-1)\), and continue. Stop at the first failure: that hypothesis and all hypotheses with larger p-values are not rejected. This sequential rule spends a less stringent threshold on later tests, while preserving family-wise error control.
Worked Example: Defining a Family and Applying Bonferroni
Suppose a hypothetical study prespecifies 120 variants, two environmental exposures, and two interaction scales for each variant–exposure pair. It uses one interaction test for each combination, giving \(m=120\times2\times2=480\) tests. At family-wise level \(\alpha=0.05\), the Bonferroni threshold is:
For a particular variant–exposure–scale test, suppose the observed p-value is \(0.000080\). Its adjusted p-value is \(480(0.000080)=0.0384\), so it is significant at the 0.05 family-wise level. A second test with p-value \(0.000120\) has adjusted p-value \(480(0.000120)=0.0576\), so it does not pass. The first result supports rejecting that test’s null while controlling the probability of at least one false rejection across the stated family at no more than 0.05, assuming each p-value is valid under its null. It does not establish that the interaction is causal or that the effect will replicate.
The family in this example is appropriate only if those 480 tests capture the analysis opportunities relevant to the reported claim. If investigators also tried additional exposure cut points and selected the most favorable one, those tests cannot be ignored simply because they were omitted from the original count.
Worked Example: Holm’s Step-Down Procedure
Consider five hypothetical p-values from one prespecified family: \(0.001, 0.008, 0.018, 0.021,\) and \(0.044\). They are already ordered. With \(\alpha=0.05\), Holm’s comparisons are:
The first two tests are rejected. The third comparison fails, so the remaining tests are not rejected under Holm’s procedure, even though their unadjusted p-values are below 0.05. The conclusion is that two hypotheses pass the family-wise procedure, not that the other three null hypotheses have been shown to be true. Holm is useful when the goal is to limit the chance of any false positive but Bonferroni’s single threshold would be unnecessarily strict for some of the smaller p-values.
Exploratory Searches and False Discovery Rate
In a broad discovery analysis, researchers may prefer to control the expected fraction of false positives among the findings rather than the chance of any false positive. The Benjamini–Hochberg (BH) procedure controls FDR at target \(q\) when the tests are independent, and under certain forms of positive dependence. For arbitrary dependence, that guarantee need not hold; a conservative alternative such as Holm controls FWER regardless of dependence, while dependence-robust FDR procedures are also available.
To apply BH, sort the \(m\) p-values in ascending order. Find the largest rank \(k\) for which \(p_{(k)}\leq kq/m\). Reject the hypotheses with ranks 1 through \(k\). An FDR target of 0.05 is not a promise that 5% of any particular set of findings are false. It controls an expectation across repetitions under the procedure’s assumptions.
Worked Example: Applying Benjamini–Hochberg
Suppose an exploratory screen produces eight hypothetical, ordered p-values: \(0.001, 0.004, 0.009, 0.018, 0.031, 0.047, 0.200,\) and \(0.610\). For target \(q=0.05\), the BH cutoffs \(kq/m\), with \(m=8\), are \(0.00625, 0.0125, 0.01875, 0.0250, 0.03125, 0.0375, 0.04375,\) and \(0.0500\). The fifth p-value satisfies its cutoff, since \(0.031\leq0.03125\); the sixth does not, since \(0.047>0.0375\). Thus \(k=5\), and the first five hypotheses are rejected by BH.
This procedure is a reasonable option when the aim is to produce a set of promising candidates for follow-up, rather than to make a claim with strong control over any false positive. The interpretation depends on the dependence conditions and the validity of the p-values. Correlation among genetic tests does not automatically invalidate every method, but it means that the chosen procedure’s assumptions should be considered rather than presumed.
Prespecified Tests and Exploratory Tests
A prespecified test is defined before results are examined, including the variant or set of variants, exposure, genotype coding, interaction scale, outcome, contrast, and decision rule. For example, a protocol might name one variant, a continuous exposure measured in stated units, an additive genotype coding, and a product-term test on the mean scale as the primary G×E hypothesis. A prespecified test does not eliminate the need to address other primary hypotheses if there are several; the planned family still needs an error-control strategy.
Exploratory tests are useful for finding patterns, but a result selected from many alternatives should be described as exploratory, even if its unadjusted p-value is small. The search can include trying multiple genotype codings, exposure transformations, cut points, or interaction scales. The number of possibilities need not be a simple product if analyses overlap, but the reporting should make clear what was explored and how findings were selected. One way to limit avoidable multiplicity is to use a prespecified global test for a multi-parameter interaction, where appropriate, rather than choosing whichever individual contrast appears strongest.
The main safeguard is not a mechanical correction applied at the end. It is a transparent plan that distinguishes confirmatory claims, exploratory screens, and follow-up analyses. Reporting effect estimates and uncertainty alongside p-values also helps readers assess magnitude and precision instead of treating a threshold crossing as the whole result.
Planning Independent Replication
Replication tests whether a discovery finding is supported in data that did not contribute to selecting it. A replication plan should define the hypothesis and success criterion before examining replication results. It should match the discovery test on the substantive details that define the interaction: the variant and allele coding, exposure definition and units, outcome, interaction contrast, and analysis scale. A mean-scale product-term interaction is not automatically the same hypothesis as an interaction on a log or risk scale. Changing a binary exposure cut point or switching from a specified genotype contrast to another can likewise change the hypothesis.
Replication does not require every design detail to be identical, but differences must not silently redefine the target. If the discovery test uses exposure per 10 units, the replication analysis should use that scaling or explicitly convert it. The same direction should be defined in terms of the same contrast. If discovery screened many hypotheses, replication should address the selected hypothesis or hypotheses, and multiplicity among those replication tests should also be controlled. Replication evidence is stronger when the analysis is independently conducted and the direction, scale, and interpretation are fixed in advance.
Worked Example: A Multiplicity-Aware Replication Plan
A hypothetical discovery analysis screens 300 variants for interaction with one prespecified continuous exposure on the mean scale. It identifies three candidates for replication. Before opening a separate replication data set, investigators specify the same three variants, additive genotype coding, exposure measured per 5 units, outcome in the same units, and the same interaction coefficient as the discovery analysis. They also prespecify a one-sided test for the discovery direction and control FWER at \(\alpha=0.05\) across the three replication tests using Bonferroni.
The replication threshold for each one-sided p-value is \(0.05/3=0.0167\). Suppose the three replication p-values, tested in their prespecified directions, are \(0.009, 0.021,\) and \(0.400\). Only the first is below 0.0167. The first candidate therefore meets this particular statistical replication criterion; the other two do not. The result supports the same directional interaction hypothesis in independent data, but it does not prove the null for the other candidates or establish causation.
The criterion depends on the alignment of the analyses. If the replication team instead dichotomized the exposure at a newly chosen threshold, changed to a risk-scale interaction, or reversed the genotype contrast, the resulting p-value would not be a direct test of the prespecified replication hypothesis. Such analyses may be scientifically informative, but should be labelled as altered or additional analyses and included in an appropriate multiplicity plan.
Why These Choices Matter
Multiple testing adjustments address a specific source of false-positive evidence; they do not correct bias, measurement error, model misspecification, or an invalid p-value. Likewise, an independent replication analysis does not rescue a poorly defined discovery hypothesis. For G×E research, the exposure and the interaction scale are part of the hypothesis itself. Clear definitions make it possible to tell whether a follow-up result confirms the original claim, tests a related claim, or explores a new one.
A useful workflow is to name the scientific claim, list all tests that can support it, select an error-control method suited to the goal, and record the exact test definition needed for replication. This separates a broad search for candidates from a focused confirmatory evaluation.
Specify variants, exposures, scales, and models that could lead to the claim; count the relevant tests.
Use FWER control for strong protection against any false positive, or an appropriate FDR procedure for a discovery list, stating its assumptions.
Fix the direction, coding, exposure definition, outcome, interaction contrast, analysis scale, and multiplicity adjustment before evaluating the new data.
Common Mistakes
- Counting only reported tests. Reporting the smallest p-value after trying several models does not make the omitted tests disappear. Define the family using the analysis opportunities that could have produced the highlighted claim.
- Confusing FDR with FWER. FDR control does not mean that the probability of any false positive is at most the target level. These procedures answer different error-control questions.
- Treating a nominally significant result as replicated. Replication requires an independently evaluated, compatible hypothesis and a criterion that accounts for the replication tests being made.
- Calling a changed interaction a direct replication. Different exposure definitions or interaction scales can represent different hypotheses. Report the change rather than implying exact confirmation.
- Interpreting non-rejection as proof of no interaction. A test that does not pass its adjusted threshold is inconclusive about the null; it is not evidence that the interaction is exactly zero.
Check Your Understanding
Use the definitions and procedures in this tutorial to answer the following.
- A study tests 40 variants against three exposures using one interaction scale. What is the number of tests in this family if exactly one test is conducted for each variant–exposure pair?
- At \(\alpha=0.05\), what Bonferroni threshold applies to the 120 tests in question 1? What does passing that threshold control?
- Explain why an FDR target of 0.05 does not mean that the probability of any false positive is at most 0.05.
- Name four features of a discovery interaction hypothesis that should be matched or explicitly reconciled in replication.
- A replication analysis changes the exposure cut point and interaction scale, then obtains a small p-value. Why should this not automatically be described as direct replication?