Why the Configuration of True Nulls Matters
A familywise error rate (FWER) concerns the probability of making at least one false rejection among a family of hypotheses. But that probability can depend on which null hypotheses are true. When every null is true, every rejection is false. When some nulls are false, rejecting those hypotheses is not an error, while rejecting even one of the remaining true nulls is. A procedure’s error rate under the first situation does not, by itself, describe its error rate under the second.
This distinction matters when a trial compares several treatments with a control. Investigators may want to make claims about individual treatment comparisons, not only decide whether the complete-null hypothesis is plausible. To assess the strength of the error guarantee, it is therefore necessary to ask which configurations of true and false null hypotheses the guarantee covers.
The probability is calculated under the data-generating distribution at \(\theta\), including any randomization used by the procedure. These definitions do not require the tests to be independent. They do require a specified family and a specified procedure; changing either can change the FWER being evaluated.
Weak and Strong Control
A procedure controls FWER at level \(\alpha\) weakly if the error rate is at most \(\alpha\) whenever the complete null is true. It controls FWER at level \(\alpha\) strongly if the error rate is at most \(\alpha\) for every parameter configuration, including every partial-null configuration.
The supremum matters because the error rate can vary even among parameter values with the same pattern of true and false hypotheses. A guarantee for a configuration means the bound holds throughout that configuration, not merely at one convenient parameter value.
How the Two Guarantees Are Related
Strong control is the more demanding guarantee: it covers all configurations, including the complete null. Thus, strong control implies weak control. The reverse implication does not follow, because controlling the error rate when every null is true does not constrain what happens when some nulls are false.
This inequality justifies the implication: if the right-hand supremum is at most \(\alpha\), the left-hand supremum must also be at most \(\alpha\). It does not justify the reverse implication; a bound on a supremum over a smaller set says nothing by itself about values outside that set.
Worked Example: A Weak Guarantee That Fails Strong Control
Consider a hypothetical procedure for two hypotheses, \(H_1\) and \(H_2\), intended to control FWER at \(\alpha=0.05\). Suppose its exact FWER is as follows under four parameter configurations.
| Configuration | True nulls | FWER |
|---|---|---|
| Complete null | \(H_1,H_2\) | 0.040 |
| \(H_1\) false, \(H_2\) true | \(H_2\) | 0.120 |
| \(H_1\) true, \(H_2\) false | \(H_1\) | 0.030 |
| Both nulls false | None | 0 |
Under the complete null, every rejection is false, so the procedure’s FWER is 0.040. Since \(0.040\le0.05\), the listed complete-null rate meets the target. If that rate is the maximum over the complete-null parameter values, this supports weak control at level 0.05.
In the configuration where \(H_1\) is false and \(H_2\) is true, only a rejection of \(H_2\) is a familywise error. The stated FWER is \(0.120\), and \(0.120>0.05\). The procedure therefore fails strong control. The complete-null result cannot rescue this failure: it addresses a different configuration.
Checking the Guarantee Across Configurations
For \(m\) hypotheses, a configuration can be described by the subset of indices that are true. There are \(2^m\) possible subsets, including the empty subset (all nulls false) and the full subset (the complete null). When no null is true, a false rejection is impossible, so the FWER is zero. Strong control must still cover every configuration with at least one true null.
The following sequence is useful for assessing a claim of control. It describes what must be checked, not a particular multiplicity-adjustment procedure.
Identify the hypotheses counted together and the rule that determines which are rejected.
For the parameter configuration being assessed, list which hypotheses are true. A false rejection can occur only within this set.
Calculate or bound \(P_\theta\{R\cap T(\theta)\ne\varnothing\}\) under the data-generating model for that configuration.
For weak control, consider complete-null configurations. For strong control, the bound must hold across all configurations.
A proof of strong control must account for all relevant configurations, or use a valid argument that covers them at once. Merely listing a few selected configurations is not enough to establish the supremum bound unless the omitted cases are also covered by the proof. Similarly, a procedure’s name or its performance in one simulation is not itself a proof of the guarantee.
Worked Example: Classifying a Strong-Control Guarantee
For a hypothetical three-hypothesis procedure, suppose an exact analysis establishes the following maximum FWER within each true-null configuration. The target is \(\alpha=0.05\).
| Number of true nulls | Largest FWER over configurations in this row |
|---|---|
| 3 | 0.044 |
| 2 | 0.049 |
| 1 | 0.050 |
| 0 | 0 |
The largest listed rate across all rows is $$ \max(0.044,0.049,0.050,0)=0.050. $$ Because this maximum includes every configuration and is no greater than \(0.05\), the procedure has strong FWER control at level 0.05, assuming the stated analysis truly covers every configuration and parameter value. It consequently has weak control as well.
If the row with one true null had instead reported a maximum FWER of 0.052, the complete-null result of 0.044 would still support weak control, but the overall maximum would exceed the target. Strong control at level 0.05 would then fail.
Using Simulation as a Diagnostic
When exact probabilities are difficult to obtain, simulation can help reveal configurations where a procedure may exceed its target. The simulation must represent the data-generating model and the testing procedure, and it must include configurations beyond the complete null if strong control is the claim. A simulation estimate is evidence about performance under the simulated settings; it is not, by itself, a proof of a uniform probability bound.
Worked Example: Interpreting Simulated FWER Estimates
Suppose a hypothetical simulation runs 10,000 trials under each of two configurations for a procedure with target \(\alpha=0.05\). In each run, investigators count whether at least one true null was rejected.
Under the complete null, 420 of 10,000 runs contain at least one rejection. The estimated FWER is $$ \widehat{\operatorname{FWER}}=\frac{420}{10{,}000}=0.0420. $$ Under a partial-null configuration, 610 of 10,000 runs contain at least one false rejection, giving $$ \widehat{\operatorname{FWER}}=\frac{610}{10{,}000}=0.0610. $$
The complete-null estimate is below 0.05, while the partial-null estimate is above it. This pattern is a warning that the procedure may have weak but not strong control. Because these are simulation estimates, sampling variability matters. For the partial-null estimate, the approximate Monte Carlo standard error is $$ \sqrt{\frac{0.0610(1-0.0610)}{10{,}000}}\approx0.00239. $$ The estimate is not an exact FWER, and checking only these two simulated settings would not establish a bound over all configurations.
Interpretation, Alternatives, and Limitations
Weak and strong control answer different questions. Weak control limits the probability of any false rejection when all hypotheses in the family are true. Strong control limits that probability even when some hypotheses are false. Neither term specifies how the testing procedure achieves its bound; they describe the scope of the error guarantee.
A complete-null analysis can be useful when the scientific question is solely whether any member of the family differs from its null value. But if the intended conclusions include identifying particular effective treatments while other treatments may be ineffective, the relevant error guarantee must also address partial-null configurations. The introductory material on familywise error rate as a probability defines the event being controlled; the weak-versus-strong distinction specifies which parameter configurations the control must cover.
Strong control can be harder to establish because the probability law and error events may change across configurations. Dependence among test statistics, nuisance parameters, model misspecification, or an incorrectly specified family can complicate the analysis. Independence is not required by the definitions, but a proof or computation must use an appropriate joint distribution. A guarantee for one fixed family also does not automatically cover additional outcomes or comparisons introduced after the family was defined.
When evaluating a claim, check whether it says “under the complete null” or “under all configurations.” Also check whether the reported quantity is a probability of at least one false rejection, rather than a per-hypothesis error probability or the probability of rejecting any hypothesis. Those quantities need not be the same.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- For a family of four hypotheses, what distinguishes the complete null from a partial-null configuration?
- A procedure has FWER 0.03 under the complete null and 0.08 when two nulls are true and two are false. At target 0.05, what can be concluded from these rates?
- Why does strong control imply weak control? Does weak control imply strong control?
- In 5,000 simulated runs under one partial-null configuration, 270 runs have at least one false rejection. Calculate the estimated FWER and compare it with a target of 0.05.
Solutions. (1) Under the complete null, all four null hypotheses are true. A partial-null configuration has at least one true null and at least one false null. (2) The complete-null rate is below 0.05, but the partial-null rate exceeds it. The rates support weak control only if the complete-null bound is valid; they show a failure of strong control. (3) Strong control bounds FWER over all parameter configurations, a set that includes the complete null, so it implies weak control. Weak control does not constrain partial-null configurations and therefore does not imply strong control. (4) The estimate is \(270/5{,}000=0.0540\), which is above 0.05. It is a simulation estimate, not proof of the exact FWER, but it flags a possible strong-control problem under that configuration.