Why the Null Configuration Matters
A treatment-selection rule can change which hypotheses are tested, how many are tested, or which critical values are used. The resulting error rate therefore depends not only on the testing rule, but also on which treatment effects are truly null. A procedure that behaves acceptably when every treatment is ineffective can still make too many false efficacy claims when some treatments work and others do not.
The relevant guarantee is strong familywise error control: the probability of at least one false rejection is bounded by the chosen significance level for every configuration of true and false null hypotheses. Checking only the configuration in which all null hypotheses are true checks a necessary special case, not the full guarantee.
The phrase “at least one” is central. If three treatment hypotheses are tested, a configuration with two true nulls contributes an error whenever either of those two is falsely rejected. A false rejection of a genuinely effective treatment’s null is not a Type I error. Conversely, selecting an arm does not make its null hypothesis false or remove it from the familywise error calculation.
A Configuration-by-Configuration Audit
To assess a proposed procedure, specify the family of hypotheses before evaluating the selection rule. For each possible set \(I_0\), identify the event that at least one hypothesis in \(I_0\) is rejected. Then bound or calculate the probability of that event under the corresponding treatment-effect configuration. A complete audit need not enumerate every possible effect size if a valid general argument covers all configurations; it must, however, cover every possible set of true hypotheses.
For \(m\) hypotheses there are \(2^m\) possible true/false patterns, including the configuration with no true nulls. That all-false configuration has FWER zero by definition, but each nonempty \(I_0\) requires attention. In practice, begin with the complete null and then check partial-null cases that could expose the rule’s weaknesses: for example, one effective arm with several ineffective arms, or several effective arms with one ineffective arm.
List every treatment hypothesis that could contribute to a confirmatory claim, including arms that may later be dropped.
State which hypotheses are true and which are false. Do not infer truth from which arms the rule selects.
For this configuration, the event is a rejection of at least one hypothesis in \(I_0\), after accounting for selection and testing.
Use the procedure’s thresholds and assumptions about the null p-values and their joint distribution. Repeat for other configurations or establish a bound that covers them all.
Worked Example: A Complete-Null Check Is Not Enough
Consider a proposed rule for three treatment hypotheses. It tests each retained arm using its unadjusted level-\(\alpha\) p-value and rejects an arm’s null if that p-value is at most \(\alpha=0.05\). Suppose, for illustration, that the rule retains all three arms. Under the complete null, imagine their p-values are identical: \(P_1=P_2=P_3=U\), where \(U\) is Uniform on \([0,1]\). Each p-value is marginally valid under its null.
Under this complete-null dependence structure, the FWER is $$ P(P_1\leq 0.05\text{ or }P_2\leq 0.05\text{ or }P_3\leq 0.05) =P(U\leq 0.05)=0.05. $$ The complete-null check appears to pass.
Now consider a partial-null configuration in which \(H_1\) is false and \(H_2,H_3\) are true. Suppose the two true-null p-values are independent Uniform random variables under this configuration. A false rejection occurs if \(P_2\leq 0.05\) or \(P_3\leq 0.05\), so $$ \operatorname{FWER} = 1-P(P_2>0.05,\ P_3>0.05) =1-(0.95)^2=0.0975. $$ Thus the FWER is \(9.75\%\), greater than \(5\%\). The rule does not have strong control.
This is a deliberately simple stress-test illustrating the logic of the audit; the configuration-dependent dependence structure is not asserted to describe a particular trial. The key point is that marginally valid p-values and a passing complete-null calculation do not, by themselves, establish control for partial nulls. A real procedure needs assumptions and a bound that apply under each configuration.
Using a Bound That Covers Every Configuration
One route to strong control is to use a multiplicity procedure whose guarantee holds for any dependence among valid p-values. For example, the Bonferroni rule tests each of the \(m\) original hypotheses at \(\alpha/m\). Assume that, under every configuration, each true-null p-value is super-uniform: for every \(t\in[0,1]\), \(P(P_i\leq t)\leq t\) whenever \(H_i\) is true. For any nonempty true subset \(I_0\), the union bound gives $$ \operatorname{FWER}(I_0) =P\left(\bigcup_{i\in I_0}\{P_i\leq\alpha/m\}\right) \leq\sum_{i\in I_0}P(P_i\leq\alpha/m) \leq |I_0|\frac{\alpha}{m} \leq\alpha. $$ This argument does not require independence. It also makes the crucial assumption explicit: each true-null p-value must remain valid in the configuration being considered.
Holm’s step-down procedure, applied across all \(m\) original hypotheses, can improve on Bonferroni while retaining strong control under the same marginal validity assumption and without requiring independence. If a selection rule later restricts which Holm rejections may be reported, the reported rejections are a subset of the original Holm rejections. That extra restriction cannot increase the probability of falsely rejecting a true null. By contrast, applying Holm only to the selected hypotheses changes the family size and thresholds, so the full-family argument does not automatically apply.
Worked Example: Keeping the Original Family in the Adjustment
A hypothetical trial has \(m=3\) confirmatory hypotheses and \(\alpha=0.05\). A selection rule retains arms 1 and 2, and their p-values are \(0.012\) and \(0.021\). The procedure uses Bonferroni thresholds based on the original three-arm family, not just the two retained arms.
The threshold for each original hypothesis is $$ \frac{\alpha}{m}=\frac{0.05}{3}=0.01667. $$ For arm 1, \(0.012\leq0.01667\), so its null is rejected. For arm 2, \(0.021>0.01667\), so its null is not rejected. Arm 3 is not tested after selection and cannot be rejected by this rule.
Under any configuration, a false rejection among the reported arms must be a rejection of a true null with a p-value at most \(0.05/3\). The union-bound calculation above therefore bounds the probability of one or more such errors by \(0.05\), whether zero, one, two, or three hypotheses are true. Selection affects which rejections are possible, but it does not license replacing \(m=3\) by the observed number retained in this particular Bonferroni argument.
A Short Justification for Full-Family Holm
The strong-control argument for Holm can be seen by focusing on the true hypotheses. Order all \(m\) p-values from smallest to largest and test sequentially at \(\alpha/m,\alpha/(m-1),\ldots,\alpha\), stopping at the first failure. Let \(m_0=|I_0|\), and let \(P_{(1,I_0)}\) be the smallest p-value among the true nulls.
If Holm rejects any true null, it must have passed every earlier step, including the step at which the smallest true-null p-value is encountered. At that step, at most \(m-m_0\) false nulls can precede it. Its denominator \(m-k+1\), where \(k\) is its rank among all p-values, is therefore at least \(m_0\). Its rejection threshold is no greater than \(\alpha/m_0\). Consequently, any false rejection implies \(P_{(1,I_0)}\leq\alpha/m_0\). By the union bound and validity of each true-null p-value, $$ P(P_{(1,I_0)}\leq\alpha/m_0) \leq\sum_{i\in I_0}P(P_i\leq\alpha/m_0) \leq m_0\frac{\alpha}{m_0}=\alpha. $$ This proves the bound for an arbitrary true subset and does not assume independence. The argument is for Holm on the original family; it is not a justification for recalculating the step-down thresholds after data-driven selection.
Worked Example: Checking the Holm Bound in a Partial-Null Case
Suppose \(m=4\), \(\alpha=0.05\), and \(H_1,H_2\) are true while \(H_3,H_4\) are false. Here \(m_0=2\). If either true null is rejected by full-family Holm, the smallest true-null p-value must be at most $$ \frac{\alpha}{m_0}=\frac{0.05}{2}=0.025. $$ Therefore, $$ \operatorname{FWER}(I_0) \leq P(P_1\leq0.025)+P(P_2\leq0.025) \leq0.025+0.025=0.05. $$ The false-null p-values may influence the ordering and can lead to conservative testing, but they cannot invalidate this bound when the two true-null p-values are valid. This calculation illustrates why the guarantee concerns the true subset, rather than the number of arms selected in one realized trial.
Diagnostics, Alternatives, and Limitations
A useful audit asks what changes when the configuration changes. Does selection use the same outcomes as testing? Does dropping an arm make the thresholds for surviving arms less stringent? Are true-null p-values valid after conditioning on the selection event, or is the proposed argument relying only on their unconditional validity? Are correlations being treated as fixed when the presence of nonzero treatment effects could change the joint distribution? Each question identifies an assumption that needs justification rather than an answer based only on the complete null.
Different adjustment strategies require different guarantees. Bonferroni and full-family Holm use marginal validity and a union-bound argument, so they do not require a specified dependence structure. A correlation-based maxT procedure can use that structure to gain power, but its calibration must justify the relevant joint distribution for the configurations covered by the claim. A selection-adjusted or conditional procedure may account more directly for the selection event, but its error guarantee must match the actual selection and testing steps. None should be declared strongly controlling merely because it controls error in a single complete-null calculation.
A diagnostic simulation can reveal a potential failure if it evaluates carefully chosen partial-null configurations as well as the complete null. But simulated rates are estimates with Monte Carlo uncertainty; a finite set of simulations is not, by itself, a proof for every configuration. Conversely, a mathematical guarantee is only as credible as its assumptions about the family, the p-values, and the implementation. Strong-control verification should report both the configurations examined and the reasoning that extends the result beyond those specific cases.
The practical consequence is that a selection rule and its multiplicity adjustment must be assessed as one complete procedure. Selection can change which hypotheses are reachable, and a particular dependence pattern can change the error probability. The safest conclusion is not that a rule “looks adjusted,” but that its assumptions and rejection events imply \(\operatorname{FWER}(I_0)\leq\alpha\) for every relevant \(I_0\).
Check Your Understanding
Answer each question using the definition of FWER and the stated assumptions.
- For four hypotheses, what is the complete-null configuration? What does a partial-null configuration require?
- A procedure has two true nulls with independent Uniform p-values and rejects either one when its p-value is at most \(0.05\). Calculate the FWER for those true hypotheses.
- With \(m=5\), \(\alpha=0.05\), and two true nulls, what upper bound does the Bonferroni union-bound argument give when all five hypotheses are tested at \(\alpha/m\)?
- Why does applying Holm to all original hypotheses and then restricting reported rejections to selected arms preserve the full-family guarantee?
- Why does a complete-null error-rate check alone not establish strong control?
Solutions. (1) The complete null has all four null hypotheses true. A partial null has at least one true and at least one false null. (2) The probability of at least one rejection is \(1-(1-0.05)^2=0.0975\), or \(9.75\%\). (3) Each true p-value is tested at \(0.05/5=0.01\), so the union bound is \(2(0.01)=0.02\), below \(0.05\). (4) Restricting the full-family rejection set can only remove rejections; it cannot add a rejection that full-family Holm did not make. (5) Strong control requires the bound for every possible true subset, and partial-null configurations can have different selection behavior or dependence from the complete null.