Why Holm Controls Strong Familywise Error

Why the Strong-Control Proof Works

Holm’s step-down procedure is often introduced as a sequence of ordered comparisons. Its important guarantee is about the whole family: across all treatment comparisons in a prespecified family, the probability of making at least one false rejection is at most the chosen significance level. The key to proving this guarantee is not to reason from the global null alone. Instead, fix any configuration of true and false null hypotheses and examine the first true null encountered in the ordered list.

This argument applies to a family of \(m\) hypotheses with p-values \(p_1,\ldots,p_m\). It requires each p-value to be valid for its own null hypothesis. It does not require the p-values to be independent.

Definition: A p-value \(p_i\) is valid under its null hypothesis if, whenever that null is true, \(P(p_i\leq u)\leq u\) for every \(u\) between 0 and 1. This condition allows conservative or discrete p-values; it does not require a uniform distribution.

Order the p-values from smallest to largest, writing \(p_{(1)}\leq p_{(2)}\leq\cdots\leq p_{(m)}\). Holm’s procedure compares \(p_{(k)}\), in turn, with \(\alpha/(m-k+1)\), starting at \(k=1\). It rejects at a position if the p-value is no larger than that position’s threshold, and stops at the first position that fails. Thus, to reject a hypothesis at rank \(k\), the procedure must have passed every comparison at ranks \(1,\ldots,k\).

Theorem: If every p-value is valid under its own null, Holm’s procedure controls the familywise error rate at level \(\alpha\): for any configuration of true and false null hypotheses, the probability of rejecting at least one true null is at most \(\alpha\). The result holds under arbitrary dependence among the p-values.

The Earliest True Null and the Bound

To establish strong control, consider any fixed configuration of the hypotheses. Let \(t\) be the number of true null hypotheses, so \(t\geq 1\) when a false rejection is possible. Among the ordered p-values, define \(j\) to be the earliest rank occupied by any true null hypothesis. This tie convention matters: if multiple p-values tie for the smallest true-null p-value, use the earliest rank occupied by a true null among those ties. Every position before \(j\) is then occupied by a false null. Since there are only \(m-t\) false nulls, \(j\leq m-t+1\).

j\leq m-t+1 \quad\Longrightarrow\quad m-j+1\geq t \quad\Longrightarrow\quad \frac{\alpha}{m-j+1}\leq\frac{\alpha}{t}.

The last inequality says that Holm’s threshold at the earliest true-null position is no larger than \(\alpha/t\). If Holm rejects any true null, it must have reached and passed rank \(j\). In particular, the p-value at rank \(j\) must satisfy the comparison at that rank. Since \(p_{(j)}\) is the smallest p-value among the true nulls, any false rejection therefore entails that at least one true-null p-value is no larger than \(\alpha/t\).

\{\text{at least one true null is rejected}\} \subseteq \left\{\min_{i\in I_0}p_i\leq\frac{\alpha}{t}\right\},

where \(I_0\) is the set of indices of the true null hypotheses and contains \(t\) hypotheses. Apply the union bound to the event on the right. The union bound says that the probability of a union of events is at most the sum of their probabilities; it does not need independence. Validity of each true-null p-value gives \(P(p_i\leq\alpha/t)\leq\alpha/t\). Therefore:

\begin{aligned} P(\text{at least one true null is rejected}) &\leq P\left(\bigcup_{i\in I_0}\left\{p_i\leq\frac{\alpha}{t}\right\}\right)\\ &\leq \sum_{i\in I_0}P\left(p_i\leq\frac{\alpha}{t}\right)\\ &\leq t\frac{\alpha}{t}=\alpha. \end{aligned}

This proves the theorem for every fixed configuration of true nulls. That is strong familywise error control. The proof does not assert that the procedure has error probability exactly \(\alpha\); the bounds can be conservative, for example when p-values are discrete or dependent in a way that makes the union bound loose.

Worked Example: Locating the Critical Rank

Consider a hypothetical family of \(m=4\) hypotheses tested at \(\alpha=0.05\). Suppose the ordered p-values are \(0.003,\ 0.009,\ 0.020,\ 0.300\), and the hypotheses at ranks 2 and 4 are true. The hypotheses at ranks 1 and 3 are false. Here \(t=2\), and the earliest true-null rank is \(j=2\).

At rank 2, Holm’s threshold is \(\alpha/(m-j+1)=0.05/(4-2+1)=0.01667\). Since \(0.009\leq0.01667\), the procedure passes that true-null comparison. It also passes rank 1, because \(0.003\leq0.05/4=0.01250\), and rank 3, because \(0.020\leq0.05/2=0.02500\). It stops at rank 4, because \(0.300>0.0500\). Thus it rejects the true null at rank 2, making a false rejection in this hypothetical set of observed results.

The bound used in the proof is visible numerically: \(j=2\leq4-2+1=3\), and the true-null p-values are \(0.009\) and \(0.300\). Their minimum is \(0.009\), which is below \(\alpha/t=0.02500\). The example illustrates the necessary event inclusion; it does not estimate the long-run error probability from one set of p-values.

Worked Example: Why the Tie Convention Matters

Suppose a hypothetical family has \(m=t=2\): both null hypotheses are true, and their p-values are tied at \(0.010\). Either ordering of the tied hypotheses gives the same p-value sequence, \(0.010,\ 0.010\). The earliest rank occupied by a true null is \(j=1\), not rank 2.

The rank bound is \(j\leq m-t+1=2-2+1=1\), which holds. Holm’s first threshold at \(\alpha=0.05\) is \(0.05/2=0.0250\), so it passes the first comparison. Its second threshold is \(0.05\), so it also passes the second comparison. In the proof, the first true-null p-value is \(p_{(1)}=0.010\), and it satisfies \(0.010\leq\alpha/t=0.0250\).

If one called rank 2 “the smallest true-null position” just because the tied p-values are equal, the rank inequality would appear to fail: \(2\not\leq1\). The correct definition selects the earliest rank occupied by a true null. The argument therefore remains valid regardless of how tied hypotheses are ordered.

Worked Example: Dependence Does Not Enter the Bound

Suppose a hypothetical family has \(m=5\) hypotheses, of which \(t=3\) are true. At \(\alpha=0.05\), the proof bounds the chance of a false rejection by considering whether any of the three true-null p-values is at most \(0.05/3=0.01667\). For each true null \(i\), validity gives \(P(p_i\leq0.01667)\leq0.01667\).

Even if the three true-null p-values are dependent, the union bound gives \(P(\text{any true-null p-value}\leq0.01667)\leq3(0.01667)=0.0500\), using the unrounded value \(0.05/3\) for the exact calculation. Holm’s false-rejection event is contained in this event, so its probability is also at most \(0.05\). Independence is not needed at any step: the proof uses marginal validity for each p-value and an upper bound on the probability of a union.

What the Guarantee Depends On

The guarantee is only as credible as the p-values and the family to which the procedure is applied. In a multi-arm trial, the family should comprise the hypotheses whose simultaneous error rate is being controlled. The p-value for each true null must be valid under that null, given the analysis actually conducted. Holm’s ordering cannot repair p-values that are anti-conservative because of an incorrect model, an unaccounted-for interim look, or an analysis chosen in a way that invalidates the p-value. Nor does the proof define which clinical questions belong in the family; that choice needs to be made for the intended inferential claim.

A useful diagnostic is to ask, for every hypothesis in the family: if this particular null were true, would its p-value satisfy \(P(p_i\leq u)\leq u\) at every relevant \(u\)? Also check that the set of hypotheses being controlled has not been narrowed after looking at the results. Dependence among valid p-values is not itself a failure of Holm’s guarantee, but invalid marginal p-values or an improperly defined family are.

The single-step Bonferroni procedure compares every p-value with \(\alpha/m\). Holm starts with that same threshold for the smallest p-value, then uses progressively larger thresholds if earlier comparisons pass. It retains strong control by the argument above and can reject additional hypotheses relative to the single-step rule. Neither procedure generally guarantees that the familywise error rate equals \(\alpha\); each guarantees it is no greater than \(\alpha\).

Common Mistakes and Limits

Using only the global-null case. Showing control when all nulls are true establishes only weak control. Strong control requires the argument to cover any configuration, including mixtures of true and false nulls.
Choosing an arbitrary tied rank. The “smallest true-null p-value” can occupy multiple tied positions. Define \(j\) as the earliest rank occupied by a true null, so all earlier ranks are false and \(j\leq m-t+1\).
Assuming independence is necessary. The union bound holds under arbitrary dependence. The required condition is validity of each p-value under its own true null.
Interpreting the bound as an observed-data probability. The theorem bounds a repeated-sampling probability under a fixed configuration of true and false hypotheses. It does not say that, after observing the p-values, the probability a particular rejection is false is at most \(\alpha\).
Expecting exact error-rate use. The result is an upper bound. It can be conservative, and it does not promise that the procedure will spend the full \(\alpha\) in every configuration.

Key takeaway: In any configuration with \(t\) true nulls, the earliest true null has rank \(j\leq m-t+1\). If Holm makes any false rejection, the smallest true-null p-value must be at most \(\alpha/t\). Marginal p-value validity and the union bound then give a familywise error probability no greater than \(\alpha\), without an independence assumption.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

For \(m=6\) hypotheses with \(t=2\) true nulls, what is the largest possible earliest true-null rank \(j\), and why?
At \(\alpha=0.05\), what threshold does the proof use for each true-null p-value when \(t=4\)?
Why does choosing the earliest rank occupied by a true null resolve the tie issue?
Does Holm’s strong-control proof require independence? State the probability inequality that justifies your answer.

Solutions. (1) The largest possible rank is \(m-t+1=6-2+1=5\), because the five positions before a later true null cannot all be false when there are only \(m-t=4\) false nulls; equivalently, \(j\leq m-t+1\). (2) The threshold is \(\alpha/t=0.05/4=0.01250\). (3) Among tied smallest true-null p-values, the earliest true rank ensures that every preceding rank is occupied by a false null, so the count of false nulls implies \(j\leq m-t+1\). (4) No. For events \(A_i=\{p_i\leq\alpha/t\}\), the union bound gives \(P(\bigcup_{i\in I_0}A_i)\leq\sum_{i\in I_0}P(A_i)\leq t(\alpha/t)=\alpha\), whether or not the p-values are dependent.

Why Holm Controls Strong Familywise Error

What You'll Learn