Multiplicity and Replication in Subgroup Analysis

Why Subgroup Findings Need Special Care

A treatment may have different effects across patient groups, and subgroup analysis can help describe that heterogeneity. But every additional contrast also creates another opportunity for a small p-value to appear by chance. If analysts search many subgroups, outcomes, or definitions and report only the most striking result, a conventional unadjusted p-value can make a chance pattern look like strong evidence.

The issue is not that subgroup analysis is inherently invalid. The issue is the relationship between the analysis that produced a finding and the evidence used to assess it. A contrast specified before examining the data has a different inferential status from one chosen because its estimated effect looked unusually large. Multiplicity adjustments address repeated testing within a defined family; independent confirmation addresses the additional selection involved in discovery.

Subgroup Contrasts and the Testing Family

For a baseline-defined subgroup \(g\), let \(\tau_g\) denote the subgroup average treatment effect: the mean outcome under treatment minus the mean outcome under control for members of that subgroup. In a randomized trial, a difference in subgroup mean outcomes estimates this contrast, with a standard error used to quantify sampling uncertainty. The null hypothesis for a particular contrast is \(H_{0g}: \tau_g=0\); a two-sided alternative is \(H_{ag}: \tau_g\ne 0\).

A testing family is the set of hypotheses treated as one inferential question. It might include treatment contrasts for every prespecified subgroup, or contrasts for several outcomes and subgroups if all are searched before a result is selected. The appropriate family depends on the claims being made and on the actual analysis process. Counting only the tests eventually reported, while ignoring the tests examined during selection, understates multiplicity.

When tests are conducted separately at level \(\alpha\), the probability of at least one false rejection generally increases with the number of tests. If \(m\) tests were independent and every null were true, that probability would be \(1-(1-\alpha)^m\). Independence is not needed for the Bonferroni guarantee: by the union bound, the probability of at least one false rejection is no greater than the sum of the individual error probabilities.

Bonferroni and Holm Procedures

Bonferroni is straightforward, but it uses the same threshold for every hypothesis. The Holm step-down procedure can be more powerful while controlling FWER under arbitrary dependence, again assuming valid individual p-values. Order the \(m\) p-values from smallest to largest, \(p_{(1)}\le\cdots\le p_{(m)}\). Compare \(p_{(1)}\) with \(\alpha/m\); if it passes, compare \(p_{(2)}\) with \(\alpha/(m-1)\), and continue. Stop at the first p-value that fails; that hypothesis and all later ones are not rejected.

The step-down rule accounts for the fact that a small p-value is tested against a stricter threshold early in the sequence. If a p-value fails, stopping is essential: later hypotheses do not receive rejection under this procedure. Holm-adjusted p-values provide an equivalent reporting approach; compare each adjusted p-value with \(\alpha\).

Another error criterion is the false discovery rate (FDR), the expected proportion of false rejections among all rejections, with the proportion defined as zero when there are no rejections. Procedures such as the Benjamini–Hochberg procedure target FDR and can be useful when many hypotheses are being screened. FDR control is not the same guarantee as FWER control: it does not bound the chance of even one false discovery by \(\alpha\). The choice should match the goal and the planned interpretation, rather than being made after seeing which method yields more findings.

Why Discovery and Testing in One Sample Can Mislead

Suppose analysts estimate treatment effects in many candidate subgroups, select the subgroup with the largest estimated effect, and then report its ordinary confidence interval or p-value as if that subgroup had been specified in advance. The selected estimate is likely to be unusually large partly because of random noise. This is a form of selection, often described as the winner’s curse. A standard interval calculated as though the subgroup had been fixed does not account for the search that selected it.

One defensible strategy is to separate discovery from confirmation. Use a discovery sample to choose a subgroup according to a stated search rule. Then, before examining confirmation outcomes, lock the subgroup definition, outcome, contrast, and direction or two-sided test. Estimate and test that single contrast in an independent confirmation sample. The confirmation estimate is not guaranteed to match the discovery estimate, but it is not selected for being extreme in that sample.

Why the Bonferroni Guarantee Holds

The Bonferroni result follows directly from the union bound. Let \(I\) be the set of indices for which the null hypothesis is true, and let \(R_j\) be the event that a true null \(j\) is rejected. A family-wise error occurs if at least one event \(R_j\), for \(j\in I\), occurs. Therefore:

The second inequality uses the condition that each true-null test has probability of rejection no greater than \(\alpha/m\). Since \(|I|\le m\), the last inequality follows. No independence assumption appears in this argument. The guarantee can be conservative because the sum may exceed the actual probability of a union, particularly when rejection events overlap.

Reporting and Common Pitfalls

A transparent subgroup analysis reports the testing family, the adjustment method, the number of tests included, and whether subgroup definitions were prespecified or selected from the data. Report effect estimates and uncertainty in context, not only adjusted p-values. A small adjusted p-value does not measure the size or importance of an effect, and a nonsignificant result does not establish that subgroup effects are equal.

• Pitfall: treating every reported subgroup as prespecified. If a subgroup was chosen after comparing many candidates, an ordinary unadjusted test in that same data does not reflect the search. Disclose the selection and use an appropriate adjustment or independent confirmation.
• Pitfall: adjusting only for the tests shown in a table. Include the relevant outcomes, contrasts, or subgroup definitions that were actually examined as part of the search behind the claims.
• Pitfall: confusing a subgroup effect with a difference between subgroup effects. Evidence that an effect differs from zero in one group but not another is not itself evidence that the two effects differ. Test the between-group contrast directly.
• Pitfall: calling an unadjusted confirmation search a replication. If the confirmation data are used to try many alternatives and only a successful one is reported, those data have become another discovery sample. Prespecify one confirmation contrast or adjust for the full confirmation family.
• Pitfall: interpreting adjustment as proof. A multiplicity procedure controls a stated error rate under its assumptions; it does not guarantee that a particular finding is true, important, or causal.

Multiplicity control and replication answer related but distinct questions. An adjustment limits a specified error rate across a set of tests. Independent confirmation asks whether a selected finding is supported by new data under a locked analysis. Both require valid individual inference: for example, randomized treatment assignment for the causal interpretation in the examples, appropriately defined baseline subgroups, and valid standard errors and p-values. In observational analyses, causal interpretations additionally require a credible identification strategy; multiplicity adjustment alone cannot supply one.