Unequal Bonferroni Alpha Allocation

Why Allocate the Error Budget Unequally?

A multi-arm trial may compare several treatments with a shared control. If each comparison is tested at the same nominal significance level, the chance of making at least one false rejection across the family can exceed the target. Bonferroni control addresses this by assigning test levels whose sum is no greater than the familywise error target. Those levels need not be equal: a trial team may choose to reserve more of the error budget for a primary comparison and less for exploratory comparisons.

An unequal allocation is a design choice, not a result extracted from the observed p-values. Its justification should come from the trial’s prespecified aims, such as which comparison is primary or which treatment decision has greater importance. The allocation determines the rejection thresholds; it does not make a less stringent threshold valid merely because a comparison is important.

Definition: Suppose a prespecified family contains $m$ null hypotheses, $H_{01},\ldots,H_{0m}$, with corresponding p-values $p_1,\ldots,p_m$. An unequal Bonferroni allocation assigns each hypothesis a nonnegative test level $\alpha_i$, where $i=1,\ldots,m$, such that $$ \sum_{i=1}^{m}\alpha_i \leq \alpha. $$ Reject $H_{0i}$ if $p_i \leq \alpha_i$. The target $\alpha$ is the maximum familywise error rate to be controlled.

The inequality permits the allocations to use less than the full budget. If all $\alpha_i$ are equal and their sum is $\alpha$, this is the usual equal Bonferroni allocation, with $\alpha_i=\alpha/m$. Unequal allocation changes the individual thresholds while retaining the same familywise error guarantee, provided the assumptions below hold.

Conditions for the Guarantee

The guarantee concerns the probability of rejecting at least one true null hypothesis in the specified family. Let $I_0$ denote the indices of the null hypotheses that are true. For each true $H_{0i}$, the p-value must be valid: under that null, for every $u$ between zero and one, $P(p_i \leq u \mid H_{0i}\text{ is true}) \leq u$. This condition allows conservative p-values as well as p-values whose null distribution is exactly uniform.

The family must be defined before the analysis, and the thresholds must be fixed independently of the observed evidence used to test the hypotheses. Allocation based on prespecified clinical priorities or design information is compatible with the argument. Choosing a larger $\alpha_i$ after seeing a promising p-value is not justified by the same argument: such data-dependent selection can invalidate the assigned-level test. The argument also does not require independence among the p-values.

Theorem: If each test has a valid p-value for its null hypothesis and the nonnegative allocations satisfy $\sum_{i=1}^{m}\alpha_i\leq\alpha$, then the familywise error rate is at most $\alpha$, under any configuration of true and false null hypotheses and under arbitrary dependence among the tests.

Here the familywise error rate is the probability of rejecting at least one true null in the family. Strong control means that the bound holds whether all, some, or none of the family’s null hypotheses are true. The justification is a direct application of the union bound:

\begin{aligned} \operatorname{FWER} &=P\left(\bigcup_{i\in I_0}\{p_i\leq\alpha_i\}\right)\\ &\leq\sum_{i\in I_0}P(p_i\leq\alpha_i)\\ &\leq\sum_{i\in I_0}\alpha_i\\ &\leq\sum_{i=1}^{m}\alpha_i\\ &\leq\alpha. \end{aligned}

The first inequality is the union bound, which does not assume independence. The second follows from p-value validity for each true null, applied at its assigned level. The third holds because the allocations are nonnegative and $I_0$ is a subset of the full family. Thus even if the tests are correlated—for example, because treatment comparisons share a control—the upper bound remains valid. It may be conservative, since the union bound does not use the joint distribution of the test statistics.

Worked Applications

Worked Example: Three Comparisons With Different Priorities

Suppose a hypothetical trial has three prespecified treatment-versus-control comparisons and a familywise target of $\alpha=0.05$. The primary comparison is assigned $0.025$, a key secondary comparison $0.015$, and an exploratory comparison $0.010$. The resulting p-values are $0.018$, $0.012$, and $0.008$, respectively. Assume each p-value is valid for its null hypothesis.

Check the budget. The allocated levels sum to $0.025+0.015+0.010=0.050$, so the budget condition is met.

Apply each threshold. For the primary comparison, $0.018\leq0.025$, so reject its null hypothesis. For the key secondary comparison, $0.012\leq0.015$, so reject its null hypothesis. For the exploratory comparison, $0.008\leq0.010$, so reject its null hypothesis.

The conclusion is that all three comparison-specific null hypotheses are rejected under this prespecified procedure. This does not mean that each rejection has a 5% error probability; each test uses its own assigned level. The familywise probability of at least one false rejection is bounded above by 5%.

Worked Example: Leaving Part of the Budget Unused

Consider a different hypothetical three-comparison family with target $\alpha=0.05$. The team assigns levels $0.020$, $0.015$, and $0.005$, leaving part of the budget unused. The corresponding valid p-values are $0.018$, $0.014$, and $0.006$.

Check the allocation. The sum is $0.020+0.015+0.005=0.040$, which is no greater than $0.050$. The unused amount is $0.050-0.040=0.010$.

Compare p-values with their own thresholds. The first comparison meets its threshold because $0.018\leq0.020$. The second meets its threshold because $0.014\leq0.015$. The third does not meet its threshold because $0.006>0.005$. Therefore, reject the first two null hypotheses and do not reject the third.

Leaving some budget unused does not violate the error-control guarantee. It can, however, make the procedure more conservative than one that uses the full target. The non-rejection of the third hypothesis is not proof that its null is true; it means only that its p-value did not cross the prespecified threshold.

Worked Example: Checking Strong Control When Some Nulls Are True

Suppose a hypothetical family contains four treatment comparisons, with allocated levels $0.020$, $0.015$, $0.010$, and $0.005$. The target is $\alpha=0.05$, and assume the first and third null hypotheses are true while the second and fourth are false. The p-values are dependent, as can occur when comparisons share a control.

The full allocation uses the target exactly: $0.020+0.015+0.010+0.005=0.050$. A familywise error occurs only if at least one of the true null hypotheses, here $H_{01}$ or $H_{03}$, is rejected. By the union bound and validity of those two p-values,

\begin{aligned} P(\text{at least one false rejection}) &=P\bigl((p_1\leq0.020)\ \text{or}\ (p_3\leq0.010)\bigr)\\ &\leq P(p_1\leq0.020)+P(p_3\leq0.010)\\ &\leq 0.020+0.010\\ &=0.030. \end{aligned}

Thus the familywise error rate in this configuration is at most 0.030, which is below the target 0.050. The levels assigned to the false null hypotheses do not add to the probability of a false rejection in this configuration, because rejecting a false null is not a familywise error. Dependence does not change the validity of the upper bound.

Choosing and Auditing an Allocation

A useful allocation begins with the scientific decision the trial is meant to support. For example, a team might assign half of the total error budget to one primary comparison and divide the remainder between two secondary comparisons. With $\alpha=0.05$, this gives $0.025$, $0.0125$, and $0.0125$. The arithmetic is simple; the important work is explaining why that priority structure matches the trial objectives and recording it before the relevant results are examined.

Unequal allocation does not automatically increase overall power. It raises the threshold for some hypotheses only by lowering it for others or by changing how much of the budget is used. The allocation may improve the chance of detecting an effect for a prioritized comparison, relative to equal allocation, while making other comparisons harder to reject. The actual power for each comparison also depends on such factors as sample size, effect size, variability, and the test used.

For comparison, equal Bonferroni allocation gives every hypothesis the same threshold, $\alpha/m$. It is transparent and treats comparisons symmetrically, but may not reflect a design in which some questions are more central than others. Testing every comparison at the full $\alpha$ without adjustment uses thresholds that generally do not control the familywise error rate at $\alpha$. More elaborate stepwise procedures can use the observed p-values in a structured way, but have their own rules and justifications; unequal fixed allocation is a different, single-step approach.

Before applying the procedure, audit the specification and calculations:

Confirm that every hypothesis included in the prespecified family has an allocation, including secondary comparisons that remain part of the family.
Check that every $\alpha_i$ is nonnegative and that the total is no greater than the stated familywise target.
Verify that each p-value corresponds to the hypothesis and test direction specified in the analysis plan.
Compare each p-value with its own $\alpha_i$; do not compare all p-values with $\alpha$ or with an average threshold.
Document the rationale for priorities and ensure that the allocations were not selected in response to unblinded results.

A particularly common error is to allocate $0.05$ to each of three comparisons because $0.05$ is the desired familywise target. The resulting sum is $0.15$, not $0.05$, so the union-bound argument gives only an upper bound of $0.15$. The correction is to choose nonnegative comparison-specific levels whose sum is at most $0.05$, then use those levels consistently. Another error is to interpret a familywise bound as a guarantee that the probability of any false rejection is exactly $\alpha$. The union bound supplies an upper bound; the actual error rate may be smaller.

Key takeaway: Unequal Bonferroni allocation preserves strong familywise error control when the family is prespecified, each p-value is valid for its null, and the fixed nonnegative comparison-specific levels sum to no more than the target. The thresholds may differ, but each rejection must be judged against its own assigned level.

Check Your Understanding

Answer each question using the definitions and guarantee above. Brief solutions follow the questions.

A family has four hypotheses and target $\alpha=0.05$. Is the allocation $(0.020, 0.015, 0.010, 0.010)$ valid for the stated target?
With levels $0.030$ and $0.020$, valid p-values are $0.028$ and $0.021$. Which null hypotheses are rejected?
Does the union-bound guarantee require the treatment-comparison p-values to be independent? Give the reason.
Why is it problematic to increase one comparison’s allocated level after seeing its p-value?

Solutions. (1) No. The sum is $0.020+0.015+0.010+0.010=0.055$, which exceeds $0.050$; the stated allocation does not establish the target bound. (2) Reject the first null because $0.028\leq0.030$; do not reject the second because $0.021>0.020$. (3) No independence is required: the union bound applies to any dependence structure. Validity of each p-value under its true null supplies the individual probability bounds. (4) The threshold would then depend on observed evidence, so the fixed-level argument no longer automatically applies. The allocation should be prespecified, or an adaptive rule must have its own error-control justification.

Unequal Bonferroni Alpha Allocation

What You'll Learn