Common Multiplicity Errors In Subgroup Claims

Why Subgroup Claims Need a Multiplicity Plan

A trial can generate several plausible efficacy claims: benefit in biomarker-positive patients, benefit in biomarker-negative patients, benefit in the all-comer population, or benefit above one of several biomarker cutpoints. Each claim may be scientifically interesting. But if investigators test many claims and highlight whichever looks strongest, the chance of at least one false-positive conclusion can exceed the nominal significance level. This is a multiplicity problem, not a special property of a particular subgroup test.

The relevant question is not simply whether a reported p-value is below 0.05. It is whether the testing procedure kept the false-positive risk under control for the set of claims that could be presented as confirmatory. That set should be defined before outcome data are examined. In subgroup analyses, the population definitions, endpoints, directions of testing, and rules for moving from one claim to another all matter.

Definition: A family of hypotheses is the set of claims for which a study intends to control the probability of making one or more false rejections. The family-wise error rate (FWER) is the probability of rejecting at least one true null hypothesis in that family. If the family contains $m$ null hypotheses $H_{01},\ldots,H_{0m}$, then $$ \mathrm{FWER}=P(\text{reject at least one true }H_{0j}). $$ A procedure controls the FWER at level $\alpha$ if this probability is no greater than $\alpha$ under the relevant null configurations.

Multiplicity control does not repair invalid individual tests. Before applying an adjustment, each p-value must be valid for its stated null under the trial design and analysis: for example, the treatment comparison, endpoint, subgroup definition, and test direction must be appropriate, and the analysis must respect randomization and the prespecified handling of missing data. The adjustments below address the additional risk from making multiple claims; they do not validate a misspecified model or an outcome-informed population definition.

How Unadjusted Subgroup Testing Inflates Error

Suppose two subgroup null hypotheses are both true, and each is tested at level $\alpha=0.05$. If the two tests were independent, the probability of no false rejection would be $0.95^2=0.9025$. Thus the chance of at least one false rejection would be $$ 1-(1-0.05)^2=0.0975. $$ The overall false-positive risk is 9.75%, not 5%. With more independent tests, the chance rises further. Subgroup tests are often correlated because patients, outcomes, or test statistics overlap, so this independence calculation is not generally the actual FWER. It illustrates the problem; it is not a valid way to estimate FWER without an independence assumption.

A simple control is the Bonferroni procedure: test each of $m$ hypotheses at level $\alpha/m$. Equivalently, multiply each p-value by $m$, capping the adjusted value at 1. Bonferroni control does not require independent tests. Its justification is the union bound:

P\left(\bigcup_{j\in I_0}\{\text{reject }H_{0j}\}\right) \leq \sum_{j\in I_0}P(\text{reject }H_{0j}) \leq |I_0|\frac{\alpha}{m} \leq \alpha,

where $I_0$ is the set of true null hypotheses. The procedure can be conservative, particularly when subgroup tests are strongly correlated, but it provides a transparent control that does not rely on knowing their joint distribution.

Holm’s step-down procedure is another general FWER-controlling option. Order the $m$ p-values from smallest to largest, $p_{(1)}\leq\cdots\leq p_{(m)}$. Compare $p_{(1)}$ with $\alpha/m$; if it passes, compare $p_{(2)}$ with $\alpha/(m-1)$, continuing in order. Stop at the first failure and do not reject that hypothesis or any remaining one. Holm’s method controls FWER under arbitrary dependence and is at least as powerful as the single-step Bonferroni procedure.

Worked Example: Two Prespecified Biomarker Subgroups

Hypothetical trial. Investigators prespecify treatment comparisons in biomarker-positive and biomarker-negative patients, with two-sided tests and a family-wise level of 0.05. Assume each subgroup p-value comes from a valid test under its own null. The positive-subgroup test has $z=2.36$, giving a two-sided p-value of approximately 0.0183; the negative-subgroup test has $z=2.04$, giving a two-sided p-value of approximately 0.0414.

With two claims, the Bonferroni per-test threshold is $0.05/2=0.025$. The corresponding adjusted p-values are $$ p_{\mathrm{positive,adj}}=\min(1,2(0.0183))=0.0366, \qquad p_{\mathrm{negative,adj}}=\min(1,2(0.0414))=0.0828. $$

The positive-subgroup result meets the family-wise criterion because $0.0183<0.025$, or equivalently $0.0366<0.05$. The negative-subgroup result does not. The controlled conclusion is evidence of a treatment effect in the prespecified positive subgroup under the tested model and endpoint. This result alone does not establish that treatment effects differ between the subgroups: a claim of heterogeneity requires an appropriate treatment-by-biomarker interaction assessment, not the contrast between one “significant” and one “not significant” result.

Post Hoc Population Definitions

A subgroup is not defined merely by the fact that a biomarker was measured. If investigators inspect outcomes and then choose a cutoff, transformation, or combination of markers that produces the most favorable result, the reported subgroup p-value behaves as though one hypothesis had been tested even though several population definitions were searched. A nominal p-value for the selected definition does not account for that search.

A finite, fully recorded set of candidate cutpoints can sometimes be handled as a family: calculate valid p-values for each prespecified candidate and adjust across all candidates considered. Bonferroni or Holm remains valid under dependence among those tests. This requires that the search space be honestly represented. If analysts also tried unrecorded cutpoints, endpoints, or analysis variants, adjusting only for the reported candidates does not control the full selection process. An outcome-selected cutoff is generally best treated as exploratory unless a valid selection-adjusted procedure was planned.

Worked Example: Selecting the Best of Four Cutpoints

Hypothetical analysis. Before examining treatment outcomes, a statistical plan lists four candidate biomarker thresholds. Valid two-sided treatment-comparison tests at those fixed thresholds produce p-values 0.008, 0.021, 0.064, and 0.190. The analysis selects the threshold with the smallest p-value for a subgroup claim. Assume the four tests are valid at their fixed thresholds; their overlap does not invalidate Bonferroni adjustment.

Because four candidate definitions were searched, the Bonferroni threshold is $0.05/4=0.0125$. The smallest p-value is 0.008, and its adjusted value is $$ p_{\mathrm{adj}}=\min(1,4(0.008))=0.032. $$ The selected-cutpoint result therefore passes the family-wise 0.05 criterion for this four-cutpoint family. The other adjusted p-values are 0.084, 0.256, and 0.760, respectively, so none passes.

This conclusion depends on the family truly containing only those four candidate thresholds and on the fixed-threshold tests being valid. If the cutoff was chosen after a broader undocumented search, multiplying by four is not a justified correction. Even with the stated adjustment, the result supports a claim only for the population and analysis represented by the tested threshold; it does not establish that the threshold is biologically optimal or independently validated.

Unplanned All-Comer Claims

An all-comer claim is a claim about the treatment effect in the full trial population, not merely another name for a biomarker subgroup result. It may be a valid confirmatory objective if it was prespecified and tested under a multiplicity plan. It is not automatically protected because the trial also had a primary subgroup analysis, nor does it become confirmatory just because an overall p-value can be calculated after the trial.

When both a subgroup and an all-comer efficacy claim are eligible for emphasis, they may belong to the same family unless a prespecified procedure controls them separately. For example, a fixed sequence can test the first hypothesis at level $\alpha$, then test the next at level $\alpha$ only if the first is rejected, stopping at the first failure. Under valid tests and the stated sequence, this gatekeeping controls FWER for the ordered family. It cannot be retroactively claimed after seeing which result was favorable.

There is also a distinction between testing an all-comer effect and generalizing an effect from an enriched trial. An enrolled biomarker-positive population does not, by itself, provide a direct randomized comparison in all biomarker groups. Any broader population conclusion requires an appropriate basis and assumptions for that target population; a nominal all-comer claim cannot be manufactured from a subgroup p-value.

Worked Example: A Nominal All-Comer Result After a Subgroup Result

Hypothetical trial. A trial reports a two-sided p-value of 0.012 for its prespecified biomarker-positive subgroup and a p-value of 0.028 for the all-comer comparison. Suppose both claims are to be treated as confirmatory and the prespecified family contains exactly these two tests. Assume both p-values are valid for their respective null hypotheses.

Under Bonferroni control at $\alpha=0.05$, each test must have $p<0.025$. The subgroup result passes because $0.012<0.025$; its adjusted p-value is $2(0.012)=0.024$. The all-comer result does not pass because $0.028>0.025$; its adjusted p-value is $2(0.028)=0.056$.

The appropriate conclusion is that the subgroup claim meets this family-wise criterion but the all-comer claim does not. Reporting the all-comer p-value as nominally below 0.05 and calling it confirmatory would ignore the stated two-claim family. If the all-comer analysis was not planned at all, its result may still be reported transparently as exploratory, but a nominal p-value alone does not support a confirmatory efficacy claim.

Diagnostics, Comparisons, and Limits

Before interpreting a set of subgroup results, check whether the population definitions and claim rules were specified before outcomes were available; whether every tested subgroup, cutoff, endpoint, and direction is accounted for; and whether the adjustment matches the intended family. A discrepancy between the planned family and the reported analyses is a warning sign. So are selective reporting of the smallest subgroup p-value, a newly chosen biomarker threshold, or an all-comer claim introduced only after examining results.

Bonferroni is straightforward and robust to dependence, but can be conservative. Holm is also dependence-robust and usually preferable when a list of tests is available because it can reject at least as many hypotheses as Bonferroni. A prespecified hierarchical or gatekeeping procedure can allocate testing opportunity according to scientific priorities and may be more efficient, but its order and stopping rules must be established in advance. These procedures address error across claims; none turns an exploratory population definition into a prespecified one, and none replaces a suitable interaction analysis when the scientific question is whether effects vary across subgroups.

Multiplicity control also has limits. It does not guarantee that a significant finding will replicate, quantify the size or clinical importance of an effect, or eliminate bias from flawed measurements, missing outcomes, or model misspecification. Adjusted p-values should be reported with effect estimates and uncertainty intervals, population definitions, and the adjustment method. A nonsignificant adjusted result is not proof of no effect, and a significant result in one subgroup with a nonsignificant result in another is not, by itself, proof of treatment-effect heterogeneity.

Key takeaway: A subgroup claim is only as confirmatory as its population definition, analysis, and multiplicity plan. Count all claims that could drive the conclusion, control the family-wise error for that family, and label unplanned or outcome-selected findings as exploratory.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

Three valid subgroup tests are conducted at level 0.05 with no adjustment. What is the Bonferroni per-test threshold for a family-wise level of 0.05?
Four prespecified subgroup tests have ordered p-values 0.006, 0.014, 0.022, and 0.040. Under Holm’s procedure at 0.05, how many hypotheses are rejected?
Investigators choose one biomarker cutoff after examining outcomes, then report its unadjusted p-value of 0.02. What key information is needed before this can support a confirmatory claim?
A treatment effect is significant in biomarker-positive patients but not significant in biomarker-negative patients. Does that establish treatment-effect heterogeneity? Why or why not?

Solutions. (1) The threshold is $0.05/3=0.0167$ (to four significant figures); each test must be evaluated at that level for single-step Bonferroni control. (2) Holm compares 0.006 with $0.05/4=0.0125$, then 0.014 with $0.05/3=0.0167$, then 0.022 with $0.05/2=0.025$, and finally 0.040 with 0.05. All four pass, so all four null hypotheses are rejected by this procedure. (3) The full set of candidate cutoffs and other analyses searched, whether those tests are valid, and a justified selection adjustment are needed; an unadjusted p-value for the selected cutoff does not account for searching. (4) No. The difference between significant and nonsignificant results is not itself a significant difference; heterogeneity requires an appropriate interaction test or other direct comparison of subgroup effects.