Why Multiplicity Arises in Pharmacogenomics
A pharmacogenomic study can ask whether a variant modifies response to several drugs, whether the association appears for more than one outcome, and whether other variants in the same region show similar evidence. Each analysis may produce a valid p-value on its own, yet the chance of seeing at least one small p-value by chance increases as more hypotheses are tested. The central task is therefore not just to choose an adjustment formula: it is to define which scientific claims belong to the same testing family.
For example, a study might prespecify one primary outcome for one drug and one variant, while also exploring several secondary outcomes and additional variants. Those analyses do not automatically form one family or several. The family boundary should follow the claims the investigators intend to make and the analysis plan fixed before examining the results—not a boundary chosen afterward because it gives a preferred result.
For example, “the effect of genotype on response” is not always a complete hypothesis. The response might be treatment failure by a fixed time, a change in a continuous measure, or an adverse reaction; the drug, genetic contrast, and model scale also matter. Earlier material on pharmacogenomic outcomes and interaction scales distinguishes these estimands. Multiplicity planning starts by stating the hypotheses precisely enough that each test corresponds to a defined claim.
Choose the Family Before Interpreting Results
A useful family definition names the domain of claims, such as all prespecified genotype-by-treatment tests across three drugs and two outcomes. If each is tested for twelve variants, the family contains \(3\times2\times12=72\) hypotheses, provided that all those combinations were planned and tested. If one primary drug–outcome–variant test is the sole confirmatory claim, it may be designated as the primary hypothesis in advance; other tests then need to be described as secondary or exploratory rather than silently omitted from the multiplicity plan.
The number of tests is not always just the number of rows in a results table. A single model can supply multiple tests for different coefficients or contrasts. Conversely, analyses that are mathematically related may still represent distinct claims. Correlation among tests does not, by itself, decide whether they belong to one family. It matters for the properties of some procedures, but the scientific scope of the intended claims determines the family.
Worked Example: Defining a Family Across Drugs, Outcomes, and Variants
Suppose investigators prespecify tests of genotype-by-treatment interaction for three drugs, two outcomes per drug, and twelve variants. Each test concerns a defined genetic contrast and a specified interaction scale. The p-values are valid for their stated null hypotheses under the chosen models. The investigators want to make confirmatory claims across the full planned set.
Count the hypotheses. There are \(3\times2\times12=72\) tests. The family is the set of these 72 prespecified interaction hypotheses, not merely the tests that later happen to have small p-values.
Choose a target. If the goal is to keep the probability of one or more false rejections in this family at no more than \(\alpha=0.05\), the target is family-wise error rate (FWER) control. Under the Bonferroni procedure, the per-test threshold is \(\alpha/m=0.05/72=0.0006944\), where \(m=72\).
A hypothetical p-value of 0.0008 does not meet this threshold. Its Bonferroni-adjusted p-value is \(\min(72\times0.0008,1)=0.0576\), so it is not significant at the 0.05 family-wise level. This does not show that the corresponding effect is absent; it says that this test does not pass the chosen family-wise error procedure.
If the investigators instead intended confirmatory claims only for one prespecified primary test, that primary family could contain one hypothesis. But they could not redefine the family as “the one test that looked strongest” after seeing the results and still interpret its unadjusted p-value as if it had been selected in advance.
Family-Wise Error Control
The family-wise error rate is the probability of rejecting at least one true null hypothesis within the family. Bonferroni control is simple: with \(m\) tests and target level \(\alpha\), reject a hypothesis if its p-value is at most \(\alpha/m\). Equivalently, multiply each p-value by \(m\) and cap the result at 1; reject if the adjusted p-value is at most \(\alpha\). Bonferroni controls FWER at no more than \(\alpha\) if each null p-value is valid, regardless of dependence among tests. It can be conservative, especially when many tests are correlated.
Holm’s step-down procedure also controls FWER under arbitrary dependence and is never less powerful than the ordinary Bonferroni procedure applied to the same family. Sort the p-values from smallest to largest, \(p_{(1)}\le\cdots\le p_{(m)}\). Compare \(p_{(1)}\) with \(\alpha/m\), then \(p_{(2)}\) with \(\alpha/(m-1)\), continuing in order. Reject in sequence until the first p-value that fails its threshold; do not reject that hypothesis or any that follow it. The family and target level must be defined before applying the rule.
Worked Example: Applying Holm to Eight Prespecified Tests
Consider eight prespecified tests across planned drug–outcome–variant combinations. Assume their p-values are valid under their respective null hypotheses, and set the family-wise level to \(\alpha=0.05\). The sorted hypothetical p-values are 0.001, 0.006, 0.012, 0.020, 0.031, 0.047, 0.20, and 0.60.
At rank 1, compare 0.001 with \(0.05/8=0.00625\): reject. At rank 2, compare 0.006 with \(0.05/7=0.007143\): reject. At rank 3, compare 0.012 with \(0.05/6=0.008333\): do not reject. Holm stops at this point, so only the first two hypotheses are rejected. Later p-values are not tested against less stringent thresholds after this failure.
The conclusion is that the first two hypotheses pass Holm’s 0.05 FWER procedure for these eight tests. It is not a statement that either null has a particular probability of being true. The adjusted analysis also does not change the estimated interaction effects; it changes the decision rule for the family of tests.
False-Discovery Rate for Broader Screens
When the aim is to identify a set of promising associations for follow-up, controlling the false-discovery rate (FDR) may be more relevant than requiring a very small probability of any false rejection. For a set of rejected hypotheses, the FDR is the expected proportion of false rejections among all rejections, with that proportion defined as zero if there are no rejections. Unlike FWER, FDR control does not promise that the chance of even one false discovery is below the target level.
The Benjamini–Hochberg procedure (BH) controls FDR at target level \(q\) when the null p-values are independent; it also applies under certain forms of positive dependence, commonly called positive regression dependence on a subset (PRDS). Sort the \(m\) p-values. Find the largest rank \(k\) such that \(p_{(k)}\le(k/m)q\). Reject the hypotheses at ranks 1 through \(k\). If no rank satisfies the inequality, reject none. Under arbitrary dependence, this procedure does not have the same general guarantee; a more conservative dependence-robust adjustment, such as the Benjamini–Yekutieli procedure, is available.
Worked Example: Applying Benjamini–Hochberg to a Variant Screen
Suppose ten prespecified tests form an exploratory screen across drug–outcome–variant combinations. The p-values are valid under their null hypotheses, and their dependence meets the conditions for BH control. Set \(q=0.05\). The sorted hypothetical p-values are 0.001, 0.004, 0.009, 0.018, 0.025, 0.041, 0.060, 0.11, 0.40, and 0.80.
The BH thresholds by rank are \((i/10)\times0.05\): 0.005, 0.010, 0.015, 0.020, 0.025, 0.030, 0.035, 0.040, 0.045, and 0.050. The first five p-values meet their respective thresholds. At rank 6, \(0.041>0.030\), and the later p-values also fail their thresholds. The largest passing rank is therefore \(k=5\), and the first five hypotheses are rejected.
These rejections are interpreted under the FDR procedure: across repeated applications under its assumptions, the expected proportion of false rejections among the rejected hypotheses is controlled at no more than 0.05. This is not a guarantee that exactly five percent of this particular set of discoveries are false, nor a 95% probability that each selected association is true.
Prespecified and Exploratory Findings
Prespecification and multiplicity adjustment solve related but different problems. A prespecified analysis records the hypothesis, outcome definition, genetic contrast, model scale, family, and decision rule before results are examined. This limits opportunities to choose a favorable analysis after seeing the data. Multiplicity adjustment addresses the error rate for the specified collection of tests. Prespecification does not make a large family disappear, and adjustment does not undo selective reporting if tests were omitted or the family was changed after results were known.
Exploratory analyses can be valuable for finding patterns across drugs, outcomes, or variants. They should be clearly labeled as exploratory, with the set of tests and any FDR or other procedure reported. A result discovered after trying many alternative outcomes, contrasts, or models is not converted into a confirmatory result merely by adjusting the final handful of reported p-values. The full selection process matters.
Common Mistakes and Reporting
- Defining a family by significance. “We tested only the variants with small p-values” is not a defensible family rule. Specify the planned set without using the observed p-values to select its membership.
- Treating separate families as free of overall multiplicity. If separate drug-specific families are each tested at 0.05, the probability of at least one false rejection across all drugs need not be 0.05. If the scientific claim spans those families, consider a study-wide procedure or explicitly state the narrower error-control scope.
- Confusing FWER and FDR. FWER concerns any false rejection; FDR concerns the expected fraction of false rejections among rejections. A procedure controlling one does not automatically provide the other guarantee.
- Assuming correlation removes the need for adjustment. Correlated variants or outcomes can affect a procedure’s efficiency and assumptions, but correlation alone does not justify ignoring the multiple claims. Bonferroni and Holm remain valid under arbitrary dependence when the individual p-values are valid.
- Reporting only adjusted p-values. State the family membership, procedure, target level, and whether the findings were confirmatory or exploratory. Report effect estimates and uncertainty as well, so the strength and scale of associations are not reduced to a significance decision.
Check Your Understanding
Answer each question by identifying the family or error-control target clearly.
- A study tests four variants for each of three drugs and two outcomes. If every combination was prespecified, how many hypotheses are in the family?
- What does Bonferroni control, and why does its guarantee not require independent tests?
- For Holm’s procedure, what happens to the tests after the first ordered p-value fails its sequential threshold?
- How does FDR differ from FWER in the kind of error rate it controls?
- Why is it not valid to define the confirmatory family after examining which drug–variant results have the smallest p-values?