Why Multiple Subgroup Tests Need a Plan
A trial may report treatment effects for several subgroups and several outcomes. Even when every treatment effect is truly zero, repeated testing makes it more likely that at least one result will appear statistically significant by chance. A small p-value from one analysis is therefore harder to interpret when many related analyses were conducted.
The first task is to define the set of hypotheses that supports a particular set of claims. For example, a report might examine treatment effects in five prespecified age groups for one primary outcome, or examine those groups across three outcomes. Whether those tests form one family or several depends on the scientific questions and the claims to be made. It should be decided in advance, not chosen after seeing which results are significant.
For subgroup \(g\), let \(\tau_g\) denote the causal treatment effect in that subgroup on the stated effect scale. A subgroup-specific null might be \(H_{0g}:\tau_g=0\). A family can contain several such nulls, nulls for multiple outcomes, or both. The precise family matters: adjusting a set of five subgroup tests does not also adjust an unreported set of outcome tests unless those tests were included in the adjustment.
Multiplicity adjustment concerns the chance of false statistical discoveries across a defined family. It does not change the estimand, repair confounding, ensure that subgroup definitions are appropriate, or establish that an effect is clinically important. Valid p-values for each hypothesis remain a prerequisite.
How the Risk Accumulates
If \(m\) independent null hypotheses are all true and each is tested at level \(\alpha\), the probability of no false rejection in any one test is \(1-\alpha\). Independence then gives a family-wise error rate of \(1-(1-\alpha)^m\). When tests are dependent, this exact formula need not hold. However, the union bound gives an upper bound that does not require independence: if each test has type I error at most \(\alpha\), the probability of at least one false rejection is at most \(m\alpha\).
Worked Example: Eight Tests Under the Global Null
Suppose a hypothetical analysis tests eight subgroup contrasts, all of which are truly null, using unadjusted tests at \(\alpha=0.05\). First suppose the tests are independent. The probability of no false positives is \(0.95^8\), so
Thus, under these assumptions, the chance of at least one false positive is about 33.7%, not 5%. If independence is not assumed, the union bound gives \(\mathrm{FWER}\leq 8(0.05)=0.40\). This bound may be conservative, but it still shows why interpreting each of eight p-values as though it were the only test can be misleading.
Family-Wise Error Control
The Bonferroni procedure controls FWER by testing each of \(m\) hypotheses at level \(\alpha/m\). Equivalently, multiply each raw p-value by \(m\) and cap the result at 1; reject when the adjusted p-value is at most \(\alpha\). The procedure follows from the union bound, so it controls FWER under arbitrary dependence among tests, provided the individual p-values are valid. Its simplicity is useful, although it can be conservative when many hypotheses are tested.
Holm’s step-down procedure also controls FWER under arbitrary dependence and is never less powerful than applying the Bonferroni cutoff to every hypothesis. Order the raw p-values from smallest to largest, \(p_{(1)}\leq \cdots \leq p_{(m)}\). Compare them in order to increasingly permissive thresholds. Continue rejecting while each comparison passes; at the first failure, stop.
Worked Example: Holm Adjustment for Five Subgroup Contrasts
Suppose a hypothetical randomized trial reports five subgroup-specific treatment contrasts for one outcome. Their raw p-values, in increasing order, are 0.004, 0.018, 0.031, 0.047, and 0.200. The family contains all five tests, and the desired FWER level is \(\alpha=0.05\). Assume each p-value is valid for its stated null.
For \(m=5\), Holm’s thresholds are \(0.05/5=0.0100\), \(0.05/4=0.0125\), \(0.05/3\approx0.0167\), \(0.05/2=0.0250\), and \(0.05/1=0.0500\). The ordered comparisons are:
The first hypothesis passes. The second does not, so the procedure stops: the second and all larger ordered p-values are not rejected. Only the hypothesis associated with the p-value 0.004 is rejected by Holm at family-wise level 0.05. It would be incorrect to reject the hypothesis at the stopping point.
For comparison, the Bonferroni cutoff for every test would be \(0.05/5=0.0100\), also rejecting only the p-value 0.004 in this example. Holm can be less conservative in other data sets because later tests receive larger thresholds after earlier hypotheses pass.
A short justification explains Holm’s FWER guarantee. Suppose \(m_0\) null hypotheses in the family are true. If Holm rejects any true null, consider the first true null in the ordered list. Every hypothesis before it is false, so at that rank at least \(m_0\) hypotheses remain. Its threshold is therefore no greater than \(\alpha/m_0\). The probability that any of the \(m_0\) true-null p-values is at most \(\alpha/m_0\) is at most
by the union bound and the validity of each true-null p-value. Thus the probability of at least one false rejection is at most \(\alpha\), regardless of dependence among the tests.
False Discovery Rate Control
When many hypotheses are examined, researchers may prefer to control the expected proportion of false findings among the rejected findings rather than the probability of making any false rejection at all. The Benjamini–Hochberg (BH) procedure is a standard FDR procedure. It is less stringent than FWER control in many settings, but its guarantee is different: it does not promise that every rejected hypothesis is true or that the probability of any false rejection is at most \(\alpha\).
To apply BH at target FDR \(q\), sort the \(m\) p-values as above and find the largest rank \(k\) for which \(p_{(k)}\leq (k/m)q\). Reject the hypotheses corresponding to the \(k\) smallest p-values. If no rank satisfies the inequality, reject none. Under independent tests, BH controls FDR at level \(q\); it also controls FDR under certain positive-dependence conditions. The dependence structure must be considered when subgroup contrasts or outcomes share participants or controls. Holm is a safer standard choice when a dependence-robust FWER guarantee is required.
Worked Example: Benjamini–Hochberg Across Six Outcomes
Consider a hypothetical study with six prespecified outcome tests, each for the same treatment comparison. Suppose the six tests are independent, their p-values are valid under their respective nulls, and the target FDR is \(q=0.05\). The ordered p-values are 0.003, 0.012, 0.021, 0.034, 0.044, and 0.200. The BH thresholds \(i q/m\), for ranks 1 through 6, are approximately 0.00833, 0.0167, 0.0250, 0.0333, 0.0417, and 0.0500.
At rank 3, \(0.021\leq0.0250\), so the condition is satisfied. At rank 4, \(0.034>0.0333\); ranks 5 and 6 also fail their respective comparisons. Thus the largest satisfying rank is \(k=3\). BH rejects the hypotheses associated with the three smallest p-values: 0.003, 0.012, and 0.021.
Under the stated independence assumption, this procedure controls the expected proportion of false discoveries among rejected hypotheses at 0.05 over repeated applications of the procedure. It does not mean there is a 5% probability that any particular rejected finding is false. Nor does it provide a 5% upper bound on the chance of at least one false positive.
Choosing the Family and the Procedure
The family should reflect the scope of the inferential claims. If the intended claim is that treatment benefits a subgroup on any of several outcomes, those subgroup-by-outcome hypotheses may belong in the same family. If outcomes answer genuinely separate questions with separate error budgets, families may be defined separately, but that choice should be justified and reported. Repeatedly partitioning tests after inspecting results can make nominal error guarantees misleading.
A useful distinction is between a subgroup-specific effect test and a test of whether effects differ between subgroups. They are different hypotheses. Finding a small p-value in one subgroup and a large p-value in another does not, by itself, establish that the subgroup effects differ. The multiplicity procedure must be applied to the hypotheses that actually support the intended claims; it cannot turn separate significance labels into a direct test of heterogeneity.
List the subgroup contrasts, outcomes, and any other tests that will support the conclusions.
State which hypotheses are being considered jointly and the desired error criterion and level.
Use Bonferroni or Holm for FWER control; consider BH for FDR control when its dependence conditions are defensible.
Identify the family and method, distinguish raw from adjusted results, and describe effect estimates and uncertainty alongside decisions.
Common Mistakes and Interpretation
- Treating each p-value as an isolated test. A p-value below 0.05 does not automatically support a claim when it was selected from many subgroup or outcome tests. State the family and the adjustment.
- Rejecting at Holm’s stopping point. When an ordered p-value first exceeds its threshold, do not reject it or any larger ordered p-value. Only earlier hypotheses that passed their thresholds are rejected.
- Calling FDR control a guarantee for each result. FDR is an expected proportion across repeated applications, not the probability that a particular finding is false.
- Assuming adjustment establishes causality. Multiplicity procedures address error from multiple testing, not confounding, selection bias, measurement problems, or an ill-defined causal effect.
- Ignoring the effect scale and size. A corrected p-value does not convey the magnitude, precision, or clinical importance of a subgroup effect. Report the estimate and an appropriate confidence interval as well.
Check Your Understanding
Answer each question using the definitions and procedures in this tutorial.
- Under independence, what is the FWER for eight true null hypotheses each tested at 0.05, and what assumption is needed for that exact calculation?
- For Holm’s procedure, what happens to the hypothesis at the first ordered p-value that exceeds its threshold and to the larger ordered p-values?
- How does the Bonferroni procedure control FWER when the tests are dependent?
- In the BH procedure, how is the number of hypotheses to reject selected from the ordered p-values?
- Why does a significant result in one subgroup and a nonsignificant result in another not, by itself, establish effect modification?