Weighted Bonferroni Procedures for Unequal Priorities

Why Weight a Bonferroni Procedure?

A family of contrasts may contain questions with different scientific importance. For example, a study might prioritize a primary treatment comparison while also reporting several secondary contrasts. Giving every test the same Bonferroni allocation is simple, but it does not reflect that priority: every contrast receives the same testing level, and hence the same critical value when their standard errors and degrees of freedom agree.

A weighted Bonferroni procedure allocates more of the total error budget to higher-priority contrasts and less to lower-priority ones. The essential safeguard is that the weights are fixed before the test results are examined and sum to one. Dependence among the contrast statistics does not invalidate the familywise guarantee.

Weights may be chosen directly or formed from nonnegative priority scores \(a_i\), provided at least one score is positive: set \(w_i=a_i/\sum_{j=1}^m a_j\). A larger weight assigns more error probability to that hypothesis. This makes its test less stringent and, for a two-sided interval, gives it a narrower critical-value multiplier. A zero weight means that the corresponding hypothesis is not tested at a positive level; its confidence interval is the whole real line.

Weighted Tests for Contrasts

For concreteness, suppose the data follow a full-rank normal linear model \(Y\sim N(X\beta,\sigma^2 I)\), with \(n\) observations, \(p\) columns in \(X\), and \(\nu=n-p>0\). Each target is an estimable contrast \(\psi_i=c_i^\mathsf{T}\beta\), with estimate \(\hat\psi_i=c_i^\mathsf{T}\hat\beta\) and estimated standard error \(SE_i=s\sqrt{c_i^\mathsf{T}(X^\mathsf{T}X)^{-1}c_i}\). Under \(H_{0i}:\psi_i=\psi_{i0}\), the statistic \(T_i=(\hat\psi_i-\psi_{i0})/SE_i\) has a \(t\) distribution with \(\nu\) degrees of freedom. This marginal result is all the weighted Bonferroni guarantee needs; the different \(T_i\)'s need not be independent.

Strong Familywise Error Control

The familywise error rate (FWER) is the probability of rejecting at least one true null hypothesis. Strong control requires that this probability be at most \(\alpha\) for every configuration of true and false hypotheses, not only when all null hypotheses are true.

Proof. Fix any configuration of hypotheses, and let \(I_0\) be the indices of the true null hypotheses in that configuration. A familywise error occurs only if at least one hypothesis in \(I_0\) is rejected. By the union bound and the size condition for each true null,

The argument holds for every possible set \(I_0\) of true nulls. It uses no independence assumption, so it applies to correlated contrasts as well. This proves strong FWER control. \(\square\)

For a valid \(p\)-value \(p_i\), meaning \(P(p_i\le u)\le u\) under \(H_{0i}\) for every \(u\in[0,1]\), the size condition follows directly: \(P(p_i\le\alpha w_i)\le\alpha w_i\). Thus the same proof applies beyond the particular \(t\)-test setting whenever each marginal test is valid at its assigned level.

Unequal-Width Simultaneous Intervals

A two-sided test at level \(\alpha w_i\) is dual to a confidence interval with coverage \(1-\alpha w_i\). In the normal linear model above, the interval for contrast \(\psi_i\) is \(\hat\psi_i\pm t_{\nu,\,1-\alpha w_i/2}SE_i\). Each individual interval has its designated marginal coverage; their simultaneous guarantee follows from the same union-bound argument.

Proof. For every positive-weight contrast, the marginal \(t\) result gives \(P(\psi_i\notin C_i)=\alpha w_i\), since the interval uses equal tails with total tail probability \(\alpha w_i\). For zero-weight contrasts, \(P(\psi_i\notin C_i)=0\), because \(C_i=\mathbb{R}\). Another application of the union bound gives

Taking complements establishes simultaneous coverage of at least \(1-\alpha\). Dependence among the estimates does not alter this bound. \(\square\)

The intervals have unequal widths for two reasons: their standard errors may differ, and their critical values differ when their weights differ. With the same degrees of freedom and standard error, a smaller positive weight gives a smaller tail probability \(\alpha w_i/2\), hence a larger critical value and a wider interval. A larger weight spends more of the familywise budget on that contrast.

Choosing and Interpreting the Weights

Weights encode priorities, not evidence from the observed contrast estimates. They can reflect a prespecified primary endpoint, scientific importance, or a planned ordering of questions. For instance, a priority score twice as large as another produces twice the allocated test level after normalization. The scores do not change the contrast estimates or their standard errors; they change only the thresholds used for inference.

Prespecification matters. If weights are chosen after seeing which contrasts have promising results, the proof above cannot simply treat the observed weights as fixed allocations: the event of selecting a large weight may be related to the event of obtaining a small \(p\)-value. A procedure designed for data-dependent selection needs a separate justification. Likewise, a weight is not a claim that its contrast is more likely to be true; it is a rule for distributing the tolerated probability of false rejection.

Common Mistake

• Wrong reasoning: “There are four tests, so assigning each a level of \(0.05\) still gives a familywise level of \(0.05\), regardless of their dependence.” The union bound gives at most \(0.20\), not \(0.05\).
• Correction: Allocate levels whose sum is at most the familywise target. With weights summing to one, the levels \(\alpha w_i\) sum to \(\alpha\), and the union-bound proof applies for any dependence structure.
• Another common confusion: A larger weight makes the interval wider because it uses more error probability. For a two-sided interval, a larger weight increases \(\alpha w_i\), decreases the critical quantile, and therefore makes the interval narrower when the standard error is fixed.