Strong Familywise Error Control

Why Error Control Must Cover More Than the Global Null

A trial may make several confirmatory claims: for example, that a treatment improves each of two outcomes, or that it benefits patients in one or more subgroups. If every claim is tested at level $\alpha$, the chance of making at least one false claim can exceed $\alpha$. This matters especially in an adaptive trial, where interim information may influence which hypotheses are pursued or how later data are analyzed. Error control concerns the whole decision procedure, including those data-dependent choices—not just the final test viewed in isolation.

The key question is not merely whether the procedure behaves acceptably when all claims are false. It is whether the probability of at least one false rejection remains bounded when any subset of the claims is false. The first requirement is called weak familywise error control; the second, stronger requirement is called strong familywise error control.

Definition: Consider a family of $m$ null hypotheses $H_{01},\ldots,H_{0m}$, and let $R$ be the set of hypotheses rejected by a specified testing procedure. At a parameter configuration $\theta$, let $I_0(\theta)$ be the indices of the hypotheses that are true. The number of false rejections is $V=\lvert R\cap I_0(\theta)\rvert$. The familywise error rate (FWER) at $\theta$ is $$ \operatorname{FWER}(\theta)=P_\theta(V\geq 1) =P_\theta\left(\bigcup_{i\in I_0(\theta)}\{H_{0i}\text{ is rejected}\}\right). $$

The probability is taken over all randomness relevant to the procedure, such as trial outcomes and any randomization used by the design. In an adaptive trial, it also reflects the fact that an interim result can determine a later action. The family of hypotheses and the rejection rules must be specified well enough to define this probability. FWER does not count rejections of false null hypotheses as errors; it counts whether one or more true null hypotheses are rejected.

Weak and Strong Control

The global null configuration is the configuration in which all $m$ null hypotheses are true. Weak control restricts the FWER at that configuration. Strong control requires a bound at every configuration in which at least one null hypothesis is true, including configurations where some hypotheses are false and others are true.

Definition: A procedure has weak FWER control at level $\alpha$ if $\operatorname{FWER}(\theta)\leq\alpha$ whenever all hypotheses in the family are true. It has strong FWER control at level $\alpha$ if $\operatorname{FWER}(\theta)\leq\alpha$ for every parameter configuration $\theta$ in which at least one hypothesis in the family is true.

Strong control includes the global null, so strong control implies weak control. The reverse implication does not follow: when some null hypotheses are false, the data distribution and the procedure’s decisions may differ from their behavior under the global null. A procedure calibrated only for the global null may therefore reject a remaining true null too often.

The level $\alpha$ is a bound on the probability of one or more false rejections over repeated use of the procedure under the specified configuration. It is not the probability that a particular rejection is false, nor is it a probability that the null hypothesis is true. Control also depends on the statistical model and the assumptions used to justify the relevant probabilities.

Worked Example: Calculating FWER Under the Global Null

Suppose two hypotheses are both true, and a procedure rejects $H_{01}$ on event $A$ and $H_{02}$ on event $B$. In a hypothetical repeated-trial calculation, assume $P(A)=0.040$, $P(B)=0.030$, and $P(A\cap B)=0.015$. Calculate the FWER.

With both null hypotheses true, any rejection is a false rejection. Thus the error event is $A\cup B$. The addition rule gives $$ \operatorname{FWER}=P(A\cup B)=P(A)+P(B)-P(A\cap B) =0.040+0.030-0.015=0.055. $$

The FWER is $0.055$, or $5.5\%$. A procedure intended to control FWER at $\alpha=0.05$ does not meet that bound at this global-null configuration. The overlap matters: simply adding $0.040$ and $0.030$ would give an upper bound of $0.070$, not the exact probability.

Why a Global-Null Check Is Not Enough

A global-null calculation considers only one configuration. Strong control must additionally consider partial-null configurations, in which some hypotheses are true and others are false. In those configurations, FWER is the probability of rejecting at least one of the true hypotheses; rejections of false hypotheses do not belong to the error event. The rejection probabilities can change across configurations, particularly when an adaptive procedure uses interim information to select a hypothesis or change its analysis.

Worked Example: Weak Control That Fails Strong Control

Consider a hypothetical two-hypothesis procedure. Under the global null, its rejection events coincide: it rejects both hypotheses on event $C$, where $P(C)=0.040$, and rejects neither otherwise. Under a configuration where $H_{01}$ is true and $H_{02}$ is false, suppose the procedure’s adaptive rule selects the first hypothesis more often, with $P(H_{01}\text{ rejected})=0.080$. Assess control at $\alpha=0.05$.

Under the global null, at least one true null is rejected exactly on $C$. Therefore $$ \operatorname{FWER}_{\text{global}}=P(C)=0.040\leq 0.05. $$ The procedure has weak control for this stated global-null probability.

Under the partial-null configuration, $H_{02}$ is false, so rejecting it is not an error. The error event is only rejection of the true $H_{01}$, giving $$ \operatorname{FWER}_{\text{partial}}=P(H_{01}\text{ rejected})=0.080>0.05. $$ Thus the stated probabilities show weak control but not strong control. Checking only the global null would miss the excess error in the partial-null configuration.

A General Sufficient Bound

One useful justification for strong control uses the union bound: the probability of a union of events cannot exceed the sum of their probabilities. Suppose that, for every true null $H_{0i}$ and every configuration in which it is true, the procedure satisfies $P_\theta(H_{0i}\text{ is rejected})\leq\alpha_i$, where the nonnegative bounds satisfy $\sum_{i=1}^m\alpha_i\leq\alpha$. Then the procedure has strong FWER control at level $\alpha$. This condition is sufficient, not necessary; it may be conservative because rejection events can overlap.

Justification: At any configuration $\theta$, the error event is the union of rejection events for the true nulls only. Therefore $$ \operatorname{FWER}(\theta) =P_\theta\left(\bigcup_{i\in I_0(\theta)}\{H_{0i}\text{ is rejected}\}\right) \leq \sum_{i\in I_0(\theta)}P_\theta(H_{0i}\text{ is rejected}) \leq \sum_{i\in I_0(\theta)}\alpha_i \leq \sum_{i=1}^m\alpha_i \leq\alpha. $$ This argument applies to every configuration, including the global null and every partial null. It does not require independence among the test statistics or rejection events.

The hypotheses and bounds in this argument must apply to the actual testing procedure. In an adaptive design, a per-hypothesis probability bound must account for any data-dependent selection or change that affects whether that true null is rejected. Using a nominal level for a final test without establishing that it remains valid after the adaptation does not establish the required bound.

Worked Example: Bounding FWER Across Configurations

A hypothetical three-hypothesis procedure guarantees, for each hypothesis that is true under a given configuration, rejection probabilities no greater than $\alpha_1=0.020$, $\alpha_2=0.015$, and $\alpha_3=0.015$, respectively. The bounds hold whether or not the other hypotheses are true. Verify a strong FWER bound, then calculate the bound when only $H_{01}$ and $H_{03}$ are true.

Across all three hypotheses, the allocated bounds sum to $$ \alpha_1+\alpha_2+\alpha_3=0.020+0.015+0.015=0.050. $$ For any configuration, the union bound says that FWER is no greater than the sum of the bounds for the true nulls, which is itself no greater than $0.050$. The procedure therefore has strong FWER control at level $0.05$, given the stated per-hypothesis guarantees.

If only $H_{01}$ and $H_{03}$ are true, only their rejections can be errors. Hence $$ \operatorname{FWER}\leq\alpha_1+\alpha_3=0.020+0.015=0.035. $$ This is an upper bound, not necessarily the exact FWER. The exact probability depends on the overlap of the two true-null rejection events.

Diagnostics, Comparisons, and Limitations

A practical review should identify the full family of claims, the possible true-null configurations, and the rejection event in each configuration. For every configuration considered, ask which nulls are true and whether any of them can be rejected. This prevents a common error: counting all rejections as errors even when some rejected hypotheses are false, or counting only errors under the global null.

Weak control is a narrower requirement than strong control. A per-hypothesis level bound combined with the union bound is one transparent route to strong control, but can be conservative because it ignores dependence and overlap. More specialized multiple-testing procedures may use the structure of a problem to obtain less conservative control. Regardless of the approach, the claim of strong control requires a justification covering every configuration with at least one true null; a single global-null calculation is not enough.

In adaptive trials, signs that the error assessment may be incomplete include an analysis plan that does not specify how interim information changes testing, a probability calculation restricted to the all-null case, or per-hypothesis guarantees that were established only for a fixed analysis but applied after data-dependent selection. These are reasons to revisit the probability argument, not proof by themselves that the procedure fails. Strong control is also a statement about repeated use under the assumed model and specified procedure; it cannot compensate for an incorrect model or an implementation that differs from the procedure being assessed.

Common mistake: “The global-null FWER is at most $0.05$, so the procedure has strong control.” Correction: That establishes only weak control. Strong control requires the same bound for every configuration in which one or more nulls are true.
Common mistake: Adding marginal rejection probabilities and calling the sum the exact FWER. Correction: The union bound gives an upper bound; the exact FWER depends on the joint occurrence of the rejection events.
Common mistake: Counting a rejection of a false null as a familywise error. Correction: FWER counts only whether at least one true null is rejected.

Check Your Understanding

Answer each question, then compare with the solutions below.

In one configuration, $H_{01}$ and $H_{02}$ are true. Their rejection events have probabilities $0.02$ and $0.04$, with probability $0.01$ that both occur. What is the FWER?
A procedure controls FWER at $0.05$ when all four null hypotheses are true. What kind of control has been established by that fact alone?
In a partial-null configuration, $H_{01}$ is true and $H_{02}$ is false. The procedure rejects each with probability $0.03$. If those are the only hypotheses, what is the FWER?
Three tests have per-hypothesis rejection-probability bounds of $0.01$, $0.02$, and $0.02$, valid whenever each corresponding null is true. What strong FWER bound follows from the union bound?

Solutions. 1. The error event is the union of the two true-null rejection events, so $0.02+0.04-0.01=0.05$; the FWER is $0.05$. 2. This establishes weak control, because the calculation covers only the global null. 3. Only rejection of $H_{01}$ is a false rejection; therefore FWER is $0.03$, regardless of the rejection probability for the false $H_{02}$. 4. The sum of the three bounds is $0.01+0.02+0.02=0.05$, so the union bound gives strong FWER control at level $0.05$.