Weighted Bonferroni Tests

Why Weight the Error Budget?

A multi-arm trial may ask whether several treatments each improve an outcome relative to control. If every hypothesis is tested at the same nominal level, the chance of at least one false rejection across the family can exceed that level. Bonferroni control addresses this by dividing the allowed familywise error across hypotheses. When the hypotheses do not have equal priority, the division need not be equal: a prespecified weighting can assign more of the error budget to comparisons that matter most for the trial’s decision.

The weights change the thresholds for individual tests, not the definition of the family or the evidence represented by each p-value. The family must be specified before the results are examined, and each p-value must be valid for its own null hypothesis. In a multi-stage trial, this means the p-value must account for the design and any interim decisions; an ordinary fixed-sample p-value is not automatically valid for a stage-wise testing rule.

Definition: Suppose a prespecified family contains \(m\) null hypotheses \(H_{01},\ldots,H_{0m}\), with corresponding p-values \(p_1,\ldots,p_m\). Choose fixed weights \(w_1,\ldots,w_m\) such that \(w_i\geq 0\) and \(\sum_{i=1}^{m} w_i=1\). At familywise significance level \(\alpha\), weighted Bonferroni rejects \(H_{0i}\) if \(w_i>0\) and \(p_i\leq \alpha w_i\).

The weight \(w_i\) is the fraction of the total error budget assigned to hypothesis \(i\). Its local significance level is \(\alpha_i=\alpha w_i\), so the local levels sum to \(\alpha\). Equal weights, \(w_i=1/m\), give the ordinary Bonferroni threshold \(\alpha/m\). A larger weight gives a less stringent threshold, while a smaller weight gives a more stringent one. Weights are not probabilities that hypotheses are true, and they do not change the hypotheses being tested.

Why the Procedure Controls Familywise Error

A p-value is valid for a null hypothesis if, whenever that null is true, \(P(p_i\leq t)\leq t\) for every \(t\) between 0 and 1. This condition is required for each hypothesis in the family. No independence assumption among the p-values is needed.

Theorem: If each \(p_i\) is valid under \(H_{0i}\), the weights are fixed and nonnegative with sum one, and each positive-weight hypothesis is rejected when \(p_i\leq\alpha w_i\), then the probability of rejecting at least one true null hypothesis is at most \(\alpha\). This is strong familywise error control: the bound holds for any configuration of true and false null hypotheses.

To see why, let \(T\) be the set of indices of the true null hypotheses. A familywise error occurs only if at least one test in \(T\) rejects. The union bound and validity of each p-value give

\begin{aligned} P(\text{at least one true null is rejected}) &=P\left(\bigcup_{i\in T}\{p_i\leq\alpha w_i\}\right)\\ &\leq \sum_{i\in T}P(p_i\leq\alpha w_i)\\ &\leq \sum_{i\in T}\alpha w_i\\ &\leq \alpha\sum_{i=1}^{m}w_i =\alpha. \end{aligned}

The final inequality follows because \(T\) is only a subset of the whole family and all weights are nonnegative. This argument does not require the p-values to be independent, which makes the method suitable when treatment comparisons share a control or are otherwise dependent. The guarantee is an upper bound; it does not assert that the familywise error rate equals \(\alpha\).

Worked Decisions With Prespecified Weights

Worked Example: Three Treatment Comparisons With Different Priorities

Suppose a hypothetical trial compares three treatments with control. The prespecified weights are \(w_1=0.50\), \(w_2=0.30\), and \(w_3=0.20\), and the familywise level is \(\alpha=0.025\). Assume the three reported p-values are valid for their respective hypotheses. The p-values are \(p_1=0.0120\), \(p_2=0.0090\), and \(p_3=0.0050\). Calculate the thresholds and decisions.

The thresholds are \(\alpha w_i\). Thus,

0.025(0.50)=0.0125,\qquad 0.025(0.30)=0.00750,\qquad 0.025(0.20)=0.00500.

For the first comparison, \(0.0120\leq0.0125\), so reject \(H_{01}\). For the second, \(0.0090>0.00750\), so do not reject \(H_{02}\). For the third, \(0.0050=0.00500\), so reject \(H_{03}\), because equality meets the rejection rule. The familywise error rate is controlled at no more than 0.025, provided the p-values and prespecified weights meet the stated conditions.

Worked Example: Computing Weighted Adjusted P-Values

Consider a hypothetical four-hypothesis family with weights \(0.40, 0.30, 0.20,\) and \(0.10\), tested at \(\alpha=0.05\). Suppose valid p-values are \(0.0010, 0.0140, 0.0120,\) and \(0.0040\), respectively. For a positive weight, define the weighted adjusted p-value as \(\min(1,p_i/w_i)\). A hypothesis is rejected when this adjusted value is at most \(\alpha\).

The adjusted values are calculated as follows:

\begin{aligned} \min(1,0.0010/0.40)&=0.00250,\\ \min(1,0.0140/0.30)&=0.0467,\\ \min(1,0.0120/0.20)&=0.0600,\\ \min(1,0.0040/0.10)&=0.0400. \end{aligned}

At level 0.05, the first, second, and fourth hypotheses are rejected; the third is not. Equivalently, the local thresholds are \(0.0200, 0.0150, 0.0100,\) and \(0.00500\), and comparing each raw p-value with its threshold gives the same decisions. The adjusted values are on the p-value scale and make it convenient to compare the weighted decisions across a range of familywise levels.

How Priorities Should Guide Weights

Weights should reflect the trial’s prespecified scientific or decision priorities, not the observed p-values. For example, a trial might assign a larger share to its primary treatment comparison and smaller shares to secondary comparisons, or give more weight to a treatment whose result would have greater clinical relevance. The rationale should be set out in the protocol or statistical analysis plan so that the allocation is interpretable and reproducible.

A larger weight gives a hypothesis a less stringent testing threshold and can increase its chance of rejection when its null is false. It does not guarantee greater power in every situation: power also depends on the treatment effect, sample size, variability, and test. Nor does assigning a larger weight make the hypothesis more important as a matter of statistical truth. It expresses a choice about how to spend a limited error budget.

Worked Example: Reallocating Weight Before Results Are Seen

A hypothetical study has two treatment comparisons and a familywise level of \(\alpha=0.05\). Its initial equal allocation gives each comparison weight 0.50 and threshold \(0.05(0.50)=0.0250\). Before enrollment, the study team specifies that the first comparison is the primary decision and assigns weights \(0.70\) and \(0.30\).

The resulting thresholds are

\alpha w_1=0.05(0.70)=0.0350,\qquad \alpha w_2=0.05(0.30)=0.0150.

Suppose the valid p-values are \(p_1=0.0320\) and \(p_2=0.0140\). The first comparison meets its weighted threshold because \(0.0320\leq0.0350\); the second also meets its threshold because \(0.0140\leq0.0150\). Under equal weights, the first would not meet the threshold \(0.0250\), while the second would. The unequal allocation therefore changes the decisions by directing more of the error budget to the prespecified primary comparison. It does not change the overall familywise error bound, which remains at most 0.05.

Zero Weights, Diagnostics, and Alternatives

The definition allows zero weights. A hypothesis assigned \(w_i=0\) receives no share of the error budget and is not rejected by this procedure, regardless of its p-value. If every hypothesis is intended to support a confirmatory claim, assigning one a zero weight is usually inconsistent with that goal. Check that all planned claims have an appropriate positive weight and that the weights sum to one before applying the rule.

Other checks concern the inputs and the analysis plan. Confirm that the family includes all hypotheses relevant to the stated familywise claim; verify that weights were fixed without reference to the observed test results; and confirm that each p-value is valid for its null under the actual trial design. In a multi-stage design, an unadjusted p-value that ignores interim monitoring or stage-dependent selection can invalidate the guarantee. Weighting cannot repair an invalid p-value, an omitted hypothesis, or an outcome-dependent choice of weights.

Ordinary Bonferroni is the special case of equal weights and is simple when hypotheses have comparable priority. Weighted Bonferroni makes that allocation unequal while retaining a transparent, dependence-robust bound; it can be conservative because the union bound need not be tight. Other multiplicity procedures may use the p-values jointly or impose a testing order, but they have their own conditions and rules. Weighted Bonferroni’s practical advantage is that each decision follows directly from a fixed threshold; its limitation is that it does not adapt to the observed pattern of p-values.

Common pitfall: Choosing weights after seeing which treatment has the smallest p-value and then applying the corresponding weighted thresholds is not the prespecified procedure justified above. The error-control proof treats the weights as fixed; data-driven choices can alter the probability of rejection. Set and document the weights before examining the results, or use a method specifically justified for the proposed adaptive allocation.

Check Your Understanding

Work through each question, then compare your reasoning with the solutions.

A family has four hypotheses and weights \(0.40, 0.30, 0.20,\) and \(0.10\). At \(\alpha=0.05\), what is the threshold for the third hypothesis?
For the weights \(0.60, 0.25,\) and \(0.15\), suppose valid p-values are \(0.020, 0.018,\) and \(0.007\), with \(\alpha=0.04\). Which hypotheses are rejected?
Why does weighted Bonferroni retain familywise error control when the p-values are dependent?
What does a zero weight imply for the corresponding hypothesis under weighted Bonferroni?

Solutions. (1) The third threshold is \(0.05(0.20)=0.0100\). (2) The thresholds are \(0.04(0.60)=0.0240\), \(0.04(0.25)=0.0100\), and \(0.04(0.15)=0.00600\). The first p-value, \(0.020\), is below its threshold, while \(0.018>0.0100\) and \(0.007>0.00600\); reject only the first hypothesis. (3) The union bound gives a familywise error probability no greater than the sum of the individual rejection probabilities for true nulls. This argument does not require independence, and the sum of the corresponding weighted levels is at most \(\alpha\). (4) It receives no error budget and is not rejected by this procedure, even if its p-value is very small.

What You'll Learn