Why Allocate Type I Error Across Populations?
An enrichment trial may be designed to support an efficacy claim in the all-comer population, in a biomarker-defined subgroup, or in either population. If both hypotheses are tested, using the full significance level for each can inflate the chance of making at least one false efficacy claim. The family of claims must therefore have a prespecified error-control strategy.
This tutorial considers two hypotheses fixed before outcomes are examined: \(H_A\), the null hypothesis of no treatment benefit in the all-comer population, and \(H_B\), the null hypothesis of no treatment benefit in the prespecified biomarker subgroup. The subgroup may be nested within the all-comer population, and the two test statistics may be correlated. The key distinction is between fixed allocation, which assigns portions of the error budget to hypotheses, and gatekeeping, which makes access to testing depend on earlier rejection decisions.
Assume each test is prespecified and its p-value is valid: under its null hypothesis, for every \(t\) between 0 and 1, \(P(p_j\leq t)\leq t\). The test must also use the planned endpoint, analysis population, direction of benefit, and statistical model or randomization procedure. In what follows, \(\alpha\) is the trial’s total type I error budget. If a two-sided test is planned, its p-value and interpretation must consistently reflect that choice; an allocation does not change the test’s sidedness.
Fixed Allocation and Its Error-Control Justification
A fixed allocation assigns nonnegative amounts \(\alpha_A\) and \(\alpha_B\) to the all-comer and subgroup hypotheses, respectively, before examining trial results. A simple procedure rejects \(H_A\) when \(p_A\leq\alpha_A\), and rejects \(H_B\) when \(p_B\leq\alpha_B\). The allocations must satisfy \(\alpha_A+\alpha_B\leq\alpha\). They may be unequal when the design gives one population’s claim greater priority, but that priority should be set prospectively.
This is a weighted Bonferroni procedure: the weights determine the portions of the total error budget given to each test. The hypotheses need not be independent. The justification is the union bound, which says that the probability of a union of events is no greater than the sum of their probabilities.
The bound applies whether one or both hypotheses are true, so it gives strong FWER control. It may be conservative, particularly when tests are positively correlated, but correlation does not invalidate it. The earlier material on closed testing addresses a different way to control multiplicity through intersection hypotheses; here the practical focus is how to divide a fixed budget or order access to it.
Worked Example: Applying a Fixed Allocation
Suppose a hypothetical trial has \(\alpha=0.05\), with \(0.03\) allocated to the all-comer claim and \(0.02\) to the biomarker-subgroup claim. The prespecified analyses produce \(p_A=0.018\) and \(p_B=0.007\). Assume both p-values come from valid tests of the stated hypotheses.
The budget check is \(0.03+0.02=0.05\), so the allocations use, but do not exceed, the family error budget. For all-comers, \(0.018\leq0.03\), so reject \(H_A\). For the subgroup, \(0.007\leq0.02\), so reject \(H_B\). The design supports both population-specific efficacy claims under the stated testing plan. These decisions do not establish that treatment effects are equal across populations.
The p-values are not probabilities that the null hypotheses are true. Each is compared with its prespecified threshold; the error-control guarantee concerns the long-run probability of one or more false rejections across the family.
Choosing the Split Before Seeing Results
An allocation encodes which claims the design is prepared to support. Equal allocation, \(\alpha_A=\alpha_B=\alpha/2\), treats the two hypotheses symmetrically. An unequal split can give one claim a less stringent threshold and the other a more stringent one. The resulting procedure still controls FWER if the sum remains within \(\alpha\), but it changes the chance of rejection under alternatives. More alpha for a hypothesis can increase its rejection probability, while reducing the budget available to the other test.
A split should not be selected by looking at the observed p-values and then assigning more alpha to whichever result is close to significance. That makes the thresholds data-dependent and is not justified by the fixed-allocation argument. Prespecification includes the hypotheses, test definitions, allocations, and the rule for translating rejections into claims. If more than these two hypotheses are in the family, the full set of allocations must be considered: the sum of all allocated levels must not exceed \(\alpha\).
Worked Example: How Priorities Change Decisions
In a second hypothetical trial, the total level is \(0.05\). The design team prespecifies \(\alpha_A=0.01\) for the all-comer claim and \(\alpha_B=0.04\) for the subgroup claim. The valid p-values are \(p_A=0.025\) and \(p_B=0.031\).
The allocations sum to \(0.01+0.04=0.05\). For all-comers, \(0.025>0.01\), so \(H_A\) is not rejected. For the subgroup, \(0.031\leq0.04\), so \(H_B\) is rejected. The trial can support the prespecified subgroup claim, but not the all-comer claim, under this procedure.
If the same p-values were assessed under an equal split, each threshold would be \(0.025\). Then the all-comer p-value would meet its threshold exactly, while the subgroup p-value would exceed its threshold. The allocations have changed which claim is supported, even though the data are unchanged. This is why the split is a design decision, not a result-driven adjustment.
Gatekeeping: Order Determines Access
A fixed-sequence gatekeeping strategy tests hypotheses in a prespecified order, using the full level \(\alpha\) at each step, and proceeds to the next hypothesis only if the current one is rejected. For example, an all-comer-first sequence tests \(H_A\) at \(\alpha\); only after rejecting \(H_A\) does it test \(H_B\) at \(\alpha\). If a test is not rejected, the sequence stops. Thus, the downstream test is not reached unless its gate is passed.
This strategy controls FWER under arbitrary dependence. To see why, consider the possible configurations. If the first hypothesis is true, any false rejection in the sequence requires rejecting that first hypothesis, an event with probability at most \(\alpha\). If the first hypothesis is false and the second is true, a false rejection requires rejecting the second hypothesis; that event is a subset of the event that the second test’s valid p-value is at most \(\alpha\), and so its probability is at most \(\alpha\). If both hypotheses are false, there is no type I error. The same reasoning applies with the order reversed.
The guarantee is not a statement that the second test has a conditional type I error rate of \(\alpha\) given that the first test was rejected. Rather, the sequence controls the unconditional probability of any false rejection. The earlier material on gatekeeping tests across all-comer and enriched populations introduces this general family of procedures; the choice of order here clarifies which population’s claim is prioritized.
Worked Example: Comparing Two Gatekeeping Orders
Consider hypothetical valid p-values \(p_A=0.032\) and \(p_B=0.008\), with \(\alpha=0.05\). Under an all-comer-first fixed sequence, compare \(p_A\) with \(0.05\): since \(0.032\leq0.05\), reject \(H_A\) and proceed. Then compare \(p_B\) with \(0.05\): since \(0.008\leq0.05\), reject \(H_B\) as well.
Under a subgroup-first sequence, \(p_B=0.008\leq0.05\), so reject \(H_B\) and proceed to test \(H_A\). Since \(p_A=0.032\leq0.05\), reject \(H_A\) too. Both sequences support both claims for these particular p-values, but they need not produce the same decisions for other data. For example, if \(p_A=0.06\) and \(p_B=0.01\), all-comer-first stops after not rejecting \(H_A\), while subgroup-first rejects \(H_B\) and then stops at \(H_A\).
The first sequence makes the all-comer claim a prerequisite for testing the subgroup claim; the second reverses that priority. A gatekeeping procedure is therefore not simply a fixed split with different numbers: it uses a decision path, and a later hypothesis may receive no test at all if an earlier gate fails.
Comparing the Strategies and Checking the Design
| Feature | Fixed allocation | Fixed-sequence gatekeeping |
|---|---|---|
| Thresholds | Each hypothesis has its own prespecified level; allocated levels sum to at most \(\alpha\). | Each reached test uses level \(\alpha\); testing stops at the first nonrejection. |
| Access to claims | Either hypothesis can be rejected regardless of the other test’s result. | A later claim is testable only after earlier hypotheses in the sequence are rejected. |
| Primary design choice | How much error budget to assign to each claim. | Which claim must be established first. |
| Dependence requirement | No independence is needed for Bonferroni control. | No independence is needed for fixed-sequence control with valid marginal tests. |
Several checks help identify an invalid or poorly matched strategy:
- Check the family. If the plan permits additional biomarker subgroups, endpoints, or treatment comparisons to support claims, their hypotheses may also require multiplicity control.
- Check the budget or sequence. For a fixed allocation, verify that all allocated levels sum to no more than \(\alpha\). For gatekeeping, verify that the order and stopping rule are unambiguous.
- Check prespecification. The population definitions and allocation or sequence must not be revised in response to observed treatment effects or p-values.
- Check test validity. A multiplicity procedure cannot repair a p-value that is invalid for its own null hypothesis, population, endpoint, or analysis plan.
Neither strategy is universally more powerful. Fixed allocation permits a claim in either population to succeed independently, but each test receives only part of the total budget. Gatekeeping can use the full level for a reached test, yet a downstream claim is unavailable if its gate fails. Both approaches can be conservative or poorly aligned with the scientific objective if the chosen priorities do not reflect which claims matter. Their error-control guarantees also do not establish treatment-effect homogeneity, clinical relevance, or transportability of a subgroup result to the full population.
Check Your Understanding
Answer each question using the stated testing plan. Solutions follow the questions.
- With \(\alpha=0.05\), a fixed allocation gives \(\alpha_A=0.02\) and \(\alpha_B=0.025\). Is the allocation valid, and what is the maximum family-wise error bound?
- For \(\alpha=0.05\), an all-comer-first sequence has \(p_A=0.07\) and \(p_B=0.01\). Which hypotheses are rejected?
- Why does the Bonferroni allocation control FWER even when the all-comer and subgroup tests are dependent?
- Under subgroup-first gatekeeping, suppose \(p_B=0.04\) and \(p_A=0.055\), with \(\alpha=0.05\). State the decisions and explain the effect of the order.
Solutions. (1) The sum is \(0.02+0.025=0.045\leq0.05\), so the allocation is valid and the FWER is bounded above by \(0.045\), hence by \(0.05\). (2) Since \(0.07>0.05\), do not reject \(H_A\); the sequence stops, so \(H_B\) is not tested and no subgroup claim is made. (3) The event of any false rejection is contained in the union of the individual rejection events for true nulls; the union bound limits its probability by the sum of their allocated levels, without requiring independence. (4) Reject \(H_B\) because \(0.04\leq0.05\), then test \(H_A\); do not reject \(H_A\) because \(0.055>0.05\). The subgroup-first order permits the subgroup claim before the all-comer claim is assessed.