Weighted Holm Testing

Why Weight a Step-Down Procedure?

A MAMS trial may include several treatment-versus-control hypotheses, such as effects on distinct outcomes or in different patient groups. Those hypotheses need not have equal scientific priority. If priorities are specified before results are examined, weights allow more of the familywise error budget to be assigned to more important hypotheses while retaining a step-down procedure that can gain power when some hypotheses are rejected.

Weighted Holm testing combines two ideas: allocate error unequally using prespecified weights, then allow the threshold for a remaining hypothesis to increase as earlier hypotheses are rejected. The procedure still applies to one fixed family of hypotheses and controls the familywise error rate under arbitrary dependence, provided the individual p-values are valid.

Definition: Let $H_1,\ldots,H_m$ be a prespecified family of null hypotheses, with valid p-values $p_1,\ldots,p_m$. Assign each hypothesis a prespecified positive weight $w_i$, normalized so that $\sum_{i=1}^m w_i=1$. Define its weighted p-value ratio as $q_i=p_i/w_i$. Order the hypotheses so that $q_{(1)}\leq\cdots\leq q_{(m)}$. At step $k$, compare $p_{(k)}$ with $$ \frac{\alpha w_{(k)}}{\sum_{j=k}^{m} w_{(j)}}. $$ Equivalently, compare $q_{(k)}$ with $\alpha/\sum_{j=k}^{m} w_{(j)}$. Reject in order while each comparison passes. At the first failure, stop and retain that hypothesis and all hypotheses later in the order.

The weights describe relative allocation, not probabilities that hypotheses are true. For example, weights $0.40, 0.30, 0.20, 0.10$ allocate four tenths of the total weight to the first hypothesis and one tenth to the fourth. The first threshold is $\alpha w_{(1)}$, because the weights remaining at the start sum to 1. After a rejection, the denominator is the weight still in the ordered family, so a later threshold can be larger.

The order is determined by $p_i/w_i$, not by raw p-values and not by the weights alone. A hypothesis with a relatively small p-value may appear later if it also has a very small weight. The weights and the family must be fixed independently of the observed p-values; otherwise the stated error-control guarantee need not apply.

Applying the Ordered Comparisons

At each step, the threshold is calculated using the weight of the hypothesis at that step and the total weight of all hypotheses not yet passed over. A rejected hypothesis leaves the remaining set; a failed comparison ends the procedure. This stopping rule is essential: a later, smaller raw p-value cannot be tested after an earlier failure unless a different procedure was specified.

Worked Example: Four Treatment Comparisons

Suppose a hypothetical trial prespecifies four treatment-versus-control hypotheses, assigns weights $0.40, 0.30, 0.20, 0.10$, and uses $\alpha=0.05$. The observed p-values are $0.012, 0.018, 0.009, 0.032$, respectively. Assume each p-value is valid under its null and the family and weights were fixed before observing results.

First calculate the ratios. For $H_1$, $0.012/0.40=0.0300$; for $H_2$, $0.018/0.30=0.0600$; for $H_3$, $0.009/0.20=0.0450$; and for $H_4$, $0.032/0.10=0.320$. The order is therefore $H_1,H_3,H_2,H_4$.

At step 1, the threshold is $0.05(0.40)/1=0.0200$. Since $0.012\leq0.0200$, reject $H_1$.

At step 2, the remaining weight is $0.20+0.30+0.10=0.60$. The threshold for $H_3$ is $0.05(0.20)/0.60=0.0167$. Since $0.009\leq0.0167$, reject $H_3$.

At step 3, the remaining weight is $0.30+0.10=0.40$. The threshold for $H_2$ is $0.05(0.30)/0.40=0.0375$. Since $0.018\leq0.0375$, reject $H_2$.

At step 4, the remaining weight is $0.10$. The threshold for $H_4$ is $0.05(0.10)/0.10=0.0500$. Since $0.032\leq0.0500$, reject $H_4$. All four null hypotheses are rejected by the procedure. This conclusion is about the prespecified tests; it does not establish that any particular treatment effect is clinically important.

Worked Example: Stopping at the First Failure

Consider three hypothetical hypotheses with weights $0.50,0.30,0.20$, p-values $0.020,0.018,0.070$, and $\alpha=0.05$. Assume valid p-values and prespecified weights. The ratios are $0.020/0.50=0.0400$, $0.018/0.30=0.0600$, and $0.070/0.20=0.350$, so the order is $H_1,H_2,H_3$.

The first threshold is $0.05(0.50)/1=0.0250$, and $0.020\leq0.0250$, so reject $H_1$. The remaining weight is $0.50$, making the second threshold $0.05(0.30)/0.50=0.0300$. Because $0.018\leq0.0300$, reject $H_2$.

The final threshold is $0.05(0.20)/0.20=0.0500$. Since $0.070>0.0500$, do not reject $H_3$, and stop. The procedure rejects the first two hypotheses and retains the third. No later hypothesis is tested after a failure; here the failure occurs at the final step.

Why the Procedure Controls Familywise Error

The familywise error rate (FWER) is the probability of rejecting at least one true null hypothesis in the family. Assume that each true null has a valid p-value, meaning $P(p_i\leq t)\leq t$ for every $t$ between 0 and 1 when $H_i$ is true. The p-values may be dependent in any way. The following argument shows why weighted Holm controls FWER at no more than $\alpha$.

Justification: Let $T$ be the fixed set of true null hypotheses and let $W_T=\sum_{i\in T}w_i$. If the procedure rejects any true null, it must reject the first true null in the ordered list; call that hypothesis $H_{(k)}$. All hypotheses before it are false nulls, and $H_{(k)}$ passes its comparison. Thus $$ \frac{p_{(k)}}{w_{(k)}}\leq \frac{\alpha}{\sum_{j=k}^{m}w_{(j)}}. $$ The weight remaining from position $k$ onward is at least $W_T$. Therefore $$ p_{(k)}\leq\frac{\alpha w_{(k)}}{\sum_{j=k}^{m}w_{(j)}} \leq\frac{\alpha w_{(k)}}{W_T}. $$ Because the hypotheses are ordered by $p_i/w_i$, $H_{(k)}$ has the smallest ratio among the true nulls. Consequently, rejecting any true null implies that at least one $i\in T$ satisfies $p_i\leq\alpha w_i/W_T$. By the union bound and p-value validity, $$ \operatorname{FWER} \leq \sum_{i\in T}P\left(p_i\leq\frac{\alpha w_i}{W_T}\right) \leq \sum_{i\in T}\frac{\alpha w_i}{W_T} =\alpha. $$ This bound does not require independence. It does require valid p-values and fixed, positive weights; it is a strong-control result because the argument applies to any fixed configuration of true and false null hypotheses.

How Weighting Compares With Other Choices

Weighted Bonferroni testing compares each $p_i$ directly with $\alpha w_i$. It is simple and controls FWER under arbitrary dependence, but it does not increase the threshold for hypotheses that remain after earlier rejections. Weighted Holm uses the same initial allocation and can use more of the available error budget at later steps. When all weights are equal, weighted Holm reduces to the ordinary Holm step-down procedure.

Worked Example: A Later Threshold Can Permit Rejection

In a hypothetical three-hypothesis family, take $\alpha=0.05$, weights $0.60,0.25,0.15$, and p-values $0.010,0.016,0.040$. Assume all p-values are valid and the family and weights were prespecified. The ratios are $0.010/0.60=0.0167$, $0.016/0.25=0.0640$, and $0.040/0.15=0.267$, so the listed order is $H_1,H_2,H_3$.

The weighted Holm thresholds are $0.05(0.60)/1=0.0300$, $0.05(0.25)/0.40=0.03125$, and $0.05(0.15)/0.15=0.0500$. Each p-value passes its threshold: $0.010\leq0.0300$, $0.016\leq0.03125$, and $0.040\leq0.0500$. Thus all three are rejected.

Under weighted Bonferroni, the respective thresholds would be $0.0300,0.0125,0.00750$. The first p-value passes, but the second and third do not. This example shows how the step-down procedure can use larger later thresholds after earlier rejections; it does not mean every dataset will yield more rejections.

The comparison with a fixed-sequence procedure is different: weighted Holm does not encode a required scientific order of hypotheses. Its order is determined by the observed weighted p-values, while the weights express prespecified relative allocation. A scientific priority order alone is not a reason to sort hypotheses by raw p-value or to change weights after results are known.

Diagnostics, Pitfalls, and Limitations

Check normalization. Confirm that every weight is positive and their sum is 1. If an initial set of positive importance scores does not sum to 1, divide each by their total before applying the stated rule.
Check the ordering calculation. Order by $p_i/w_i$, retaining the corresponding hypothesis and weight at every step. Sorting raw p-values instead can change the procedure.
Check the denominator at each step. It is the sum of weights from the current position through the final position, not the sum of weights already rejected.
Stop at a failure. Continuing to compare later p-values after a failed step is not weighted Holm testing.
Prespecify the family and weights. Choosing weights to favor hypotheses with promising observed p-values, or redefining the family after seeing results, undermines the guarantee.

Weighted Holm adjusts testing thresholds, not the scientific meaning or validity of the individual p-values. It cannot repair a misspecified analysis or an invalid p-value. It also cannot decide which outcomes or treatment comparisons belong in the family; that definition and the justification for the weights are design decisions. The weighted thresholds may be conservative, as they rely on Bonferroni-type bounds to protect against arbitrary dependence.

Key takeaway: With positive weights fixed in advance and summing to 1, order hypotheses by $p_i/w_i$, compare sequentially with $\alpha w_{(k)}/\sum_{j=k}^{m}w_{(j)}$, and stop at the first failure. The procedure strongly controls FWER at level $\alpha$ when the individual p-values are valid.

Check Your Understanding

Work through each question, then compare your reasoning with the solutions below.

For weights $0.50,0.30,0.20$ and p-values $0.015,0.012,0.040$, which hypothesis is first in the weighted Holm order?
With weights $0.60,0.25,0.15$ and $\alpha=0.05$, what is the step-2 threshold if the first ordered hypothesis has been rejected?
At a step, the observed p-value is greater than its weighted Holm threshold. What happens to hypotheses later in the order?
Why does the FWER justification not require independent p-values?

Solutions. (1) The ratios are $0.015/0.50=0.0300$, $0.012/0.30=0.0400$, and $0.040/0.20=0.200$; the first hypothesis is first. (2) The remaining weight is $0.25+0.15=0.40$, so the threshold is $0.05(0.25)/0.40=0.03125$. (3) Stop; retain the hypothesis that failed and all hypotheses later in the order. (4) The error bound uses the union bound and validity of each true-null p-value, not a factorization of joint probabilities.

What You'll Learn