Why the Configuration of True Nulls Matters
A testing procedure may be evaluated in a situation where every null hypothesis is true, or in situations where some nulls are false and others are true. Those are different settings for familywise error. The distinction matters in genomic analyses because a family can contain many hypotheses, and a procedure may reject some hypotheses that are genuinely false while also making a false rejection among the true ones.
The familywise error rate (FWER) is the probability of making at least one false rejection in the family. To define it precisely, let \(H_1,\ldots,H_m\) be the hypotheses in a fixed family, and let \(I_0(\theta)\) be the indices of the hypotheses that are true under a data-generating configuration \(\theta\). Let \(R(\theta)\) be the set of hypotheses rejected by the procedure. The FWER at that configuration is the probability that the rejection set includes at least one true null:
The probability is taken over repeated data sets generated under the configuration \(\theta\), with the same testing procedure applied each time. If there are no true null hypotheses, then \(I_0(\theta)\) is empty and the FWER is zero: there is no true null available to reject falsely. By contrast, when all \(m\) null hypotheses are true, the FWER is the probability of rejecting at least one hypothesis in the entire family.
The complete null is the configuration in which all hypotheses in the family are true. A partial-null configuration has at least one true null and at least one false null. Strong control includes both the complete null and every partial-null configuration with at least one true null. The bound is ordinarily written as a supremum over configurations: strong control requires \(\sup_{\theta}\operatorname{FWER}(\theta)\le\alpha\). Configurations with no true null contribute zero to that supremum.
Calculating FWER for a Particular Configuration
The FWER is not the expected number of false rejections. It is the probability of one or more false rejections. If a data set produces several rejections, the family has still made just one familywise error if at least one of those rejected hypotheses is true.
Worked Example: Counting Familywise Errors in a Partial-Null Configuration
Consider a hypothetical family with two hypotheses. In the configuration being studied, \(H_1\) is false and \(H_2\) is true. Suppose ten equally likely hypothetical repetitions of the procedure produce the following rejection patterns. The counts are stipulated as the sampling distribution for this example.
| Rejection pattern | Number of repetitions | Does this include a false rejection? |
|---|---|---|
| Reject both \(H_1\) and \(H_2\) | 2 | Yes: \(H_2\) is true |
| Reject \(H_1\) only | 3 | No |
| Reject \(H_2\) only | 1 | Yes: \(H_2\) is true |
| Reject neither | 4 | No |
The true-null index set is \(I_0=\{2\}\). A familywise error occurs in the two repetitions that reject both hypotheses and in the one that rejects \(H_2\) only. Therefore,
The three repetitions that reject \(H_1\) only do not count as familywise errors, because \(H_1\) is false in this configuration. The total number of repetitions with any rejection is \(6\), giving \(6/10=0.60\), but that is not the FWER. The estimated FWER for this stipulated configuration is \(0.30\), based on the fraction of repetitions with at least one rejection of a true null. This example illustrates calculation, not a claim that the procedure controls error at any particular level.
Why Weak Control Does Not Guarantee Strong Control
Strong control implies weak control: if the error bound holds for every configuration, it holds in particular when all nulls are true. The converse does not follow. A procedure might behave well when every null is true but use evidence against a false null in a way that increases the chance of rejecting another, true null.
The following deliberately simple construction makes the distinction explicit. Take two hypotheses and a level \(\alpha\), where \(0<\alpha<1\). Suppose the procedure receives two p-values, \(p_1\) and \(p_2\), and rejects both hypotheses if at least one p-value is at most \(\alpha\); otherwise it rejects neither. For this hypothetical example, stipulate these joint sampling distributions:
- If both null hypotheses are true, \(p_1=p_2=U\), where \(U\) is uniformly distributed on \([0,1]\).
- If \(H_1\) is false and \(H_2\) is true, \(p_1=0\) and \(p_2=V\), where \(V\) is uniformly distributed on \([0,1]\).
- If \(H_1\) is true and \(H_2\) is false, \(p_1=V\) and \(p_2=0\), where \(V\) is uniformly distributed on \([0,1]\).
- If both nulls are false, \(p_1=p_2=0\).
In each configuration, any p-value corresponding to a true null is uniform, and therefore valid for its own null. Nevertheless, the familywise behavior of the stated procedure differs across configurations.
Worked Example: A Procedure With Weak but Not Strong Control
First consider the complete-null configuration. Both hypotheses are true, and the p-values equal \(U\). The procedure rejects both precisely when \(U\le\alpha\). Any rejection is a false rejection here, so
Thus, under the stipulated complete-null distribution, the procedure has weak control at level \(\alpha\). Now consider the partial-null configuration in which \(H_1\) is false and \(H_2\) is true. Since \(p_1=0\), the condition that at least one p-value is at most \(\alpha\) is always met. The procedure therefore always rejects \(H_2\), which is true:
For any \(\alpha<1\), this exceeds \(\alpha\). The procedure controls FWER under the complete null but fails strong control. The failure is not repaired by the fact that each true-null p-value is individually uniform: the rule uses a small p-value for the false null to trigger rejection of the true null.
This construction is intentionally poor as a testing rule; its purpose is to show that all-null calibration alone cannot establish a guarantee under partial-null configurations. In particular, observing or simulating a low FWER when every null is true says nothing by itself about what happens when signals are present. In a genomic setting, those signals correspond to false nulls, while some other hypotheses in the family may remain true.
A Strong Guarantee and Its Scope
A strong-control claim is a statement about the procedure over a set of possible configurations, not just one observed data set. To establish it, one must justify the FWER bound whenever any subset of hypotheses is true. This can be challenging because the joint behavior of test statistics or p-values may change as the configuration changes.
Worked Example: Verifying Strong Control for a Simple Procedure
Consider a hypothetical randomized procedure for a family of two hypotheses. At every configuration, it draws \(U\sim\operatorname{Uniform}(0,1)\). If \(U\le\alpha\), it rejects both hypotheses; otherwise it rejects neither. Assume the distribution of \(U\) is uniform at every configuration.
If both nulls are true, rejecting both creates a familywise error, so the FWER is \(P(U\le\alpha)=\alpha\). If exactly one null is true, that true null is rejected whenever \(U\le\alpha\), again giving FWER \(\alpha\). If neither null is true, no rejection can be a false rejection, so the FWER is zero. Thus the largest FWER across the four configurations is \(\alpha\), and this procedure has strong control at level \(\alpha\).
This example is a verification of the definition, not a recommended genomic testing method. The procedure treats the two hypotheses together and may fail to reject a genuinely false null on most repetitions. Error control alone does not guarantee useful power, scientific relevance, or precise localization of a signal.
Diagnostics, Comparisons, and Limitations
A practical diagnostic for a claimed strong guarantee is to ask which configurations were checked or covered by the argument. A simulation that generates data only under the complete null assesses weak control for that simulated setting. It cannot diagnose inflation that occurs only when some nulls are false. For a simulation-based assessment of strong control, include partial-null configurations with different subsets of true hypotheses, and examine the probability of at least one rejection among those true hypotheses in each configuration. The simulated estimate is subject to Monte Carlo uncertainty, so a small estimated FWER is not, on its own, a proof of a bound.
Dependence also matters to an assessment. Genomic tests can be correlated, and the joint distribution of test statistics may affect the chance of at least one false rejection. A diagnostic based on an unrealistic dependence structure may not represent the intended data-generating setting. For a formal guarantee, the assumptions and argument must cover the configurations and joint behavior for which the guarantee is claimed.
| Property | Weak FWER control | Strong FWER control |
|---|---|---|
| Configurations covered | Only the complete null | Every configuration of true and false nulls |
| Implication | Does not guarantee control under partial nulls | Implies weak control |
| What an all-null simulation can assess | Directly relevant to the stated all-null setting | Insufficient on its own |
| Typical interpretation of a level-\(\alpha\) claim | At most \(\alpha\) chance of any false rejection when all nulls hold | At most \(\alpha\) chance of any false rejection regardless of which nulls hold |
Neither guarantee should be confused with controlling the error probability of each individual test, nor with controlling the expected number of false rejections. The FWER focuses on whether the family contains at least one false rejection. Also, the guarantee applies to the specified testing family and procedure; changing what is tested or how the procedure is applied changes the claim that needs justification.
The main limitation of strong control is that it is a demanding guarantee: a proof or reliable assessment must address partial-null configurations, which may be numerous. Weak control can still be a meaningful property when the scientific question is specifically about the complete null, but it does not support a general claim of familywise protection when some hypotheses may be false. The relevant guarantee should therefore match the inferential claim being made.
Common Mistake
Wrong reasoning: “The procedure had a familywise error rate of \(0.05\) when all null hypotheses were true, so its familywise error rate is controlled at \(0.05\) in general.”
Correction: That result supports weak control only, and only under the complete-null conditions assessed. Strong control requires an FWER bound for every configuration of true and false nulls. A false null can affect the behavior of the procedure in ways that increase rejection of other, true nulls, as the two-hypothesis construction demonstrates.
Check Your Understanding
For each question, identify the relevant true-null configuration before interpreting the error rate. Solutions are included for checking your reasoning.
- A family has three hypotheses, of which \(H_1\) and \(H_3\) are true. One repetition rejects \(H_1\) and \(H_2\), but not \(H_3\). Is this a familywise error? Solution: Yes. Since \(H_1\) is true and rejected, at least one true null is rejected. The false rejection of \(H_2\) does not cancel that error.
- A procedure has FWER \(0.04\) when all five null hypotheses are true. What type of control has been established from this fact alone? Solution: At most weak control at level \(0.04\) for the complete-null setting that was evaluated. This fact alone does not establish strong control.
- Suppose strong FWER control holds at level \(\alpha\). What follows about the complete-null configuration? Solution: The complete null is one of the configurations covered by strong control. Therefore its FWER is also at most \(\alpha\), so weak control follows.
- In the two-hypothesis construction, under \(H_1\) false and \(H_2\) true, why is the FWER one? Solution: The stipulated value \(p_1=0\) always satisfies \(p_1\le\alpha\). The procedure therefore always rejects both hypotheses, including the true \(H_2\). Thus the probability of rejecting at least one true null is \(1\).
- Why is the probability of any rejection not necessarily the FWER? Solution: A rejection is a familywise error only if at least one rejected hypothesis is true. When some hypotheses are false, rejecting only false nulls produces rejections but no familywise error.