Weak and Strong FWER Under Arm Selection

Why the Null Configuration Matters

In a multi-arm multi-stage trial, interim data may determine which treatment arms continue to later stages. The final family of tests is therefore not merely a set of tests applied to a fixed list of arms: an arm must be selected to remain eligible for rejection. Error control must account for that selection and for the possibility that some arms are effective while others are not.

The central distinction is between controlling the chance of any false rejection when all treatment null hypotheses are true, and controlling it for every possible mixture of true and false null hypotheses. These are different guarantees. A procedure that behaves acceptably when every arm is ineffective can still make too many false rejections when an effective arm influences which other arms are selected.

Definition: Consider a prespecified family of $m$ arm-specific null hypotheses $H_1,\ldots,H_m$. At parameter value $\theta$, let $I_0(\theta)$ be the indices of the true null hypotheses. Let $S_i$ denote the event that arm $i$ is selected to continue, and let $R_i$ denote the event that the procedure rejects $H_i$. If rejection is possible only for a selected arm, $R_i$ includes the requirement $S_i$. The familywise error rate (FWER) at $\theta$ is $$ \operatorname{FWER}(\theta)=P_\theta\left(\bigcup_{i\in I_0(\theta)}R_i\right). $$ The procedure has weak FWER control at level $\alpha$ if this probability is at most $\alpha$ whenever all $m$ null hypotheses are true. It has strong FWER control at level $\alpha$ if it is at most $\alpha$ for every parameter value, including configurations with any mixture of true and false null hypotheses.

The probability is taken over the full trial process under the stated parameter value: outcomes at all stages, any randomization, interim selection, and final testing. The family and the meaning of each null hypothesis must be specified before applying the definition. In this tutorial, a false rejection means rejecting a true arm-specific null hypothesis; rejecting a false null is not a familywise error.

The global null is the configuration in which every arm-specific null is true. The global-null FWER is thus the probability of at least one rejection when every rejection would be false. Under a partial-null configuration, some null hypotheses are true and others false; only rejection of a true null counts as an error. Strong control requires the bound in both settings, and in every other configuration.

The Relationship Between Weak and Strong Control

Strong control includes the global-null case as one of its required configurations. Consequently, strong control always implies weak control at the same level. The converse does not follow: verifying one probability, under the global null, says nothing by itself about the error probabilities under partial-null configurations.

\text{Strong control: }\sup_{\theta}\operatorname{FWER}(\theta)\leq\alpha \quad\Longrightarrow\quad \text{Weak control: }\operatorname{FWER}(\theta)\leq\alpha \text{ when all }H_i\text{ are true.}

This implication is a direct consequence of the definitions. The parameter values with all nulls true are among the parameter values covered by the strong-control requirement. To demonstrate that the reverse implication is not automatic, it is enough to give a procedure whose global-null error is below the target but whose error under some partial-null configuration exceeds it.

Worked Example: A Global-Null Guarantee Is Not a Strong Guarantee

Consider a hypothetical two-arm procedure with target $\alpha=0.05$. Suppose its exact operating probabilities are summarized by the following finite probability model: within each configuration, 100 equally likely outcomes represent the full trial process, including selection and final testing.

Configuration	True nulls	Outcomes with at least one true null rejected	FWER
Both nulls true	$H_1,H_2$	2 of 100	$2/100=0.02$
$H_1$ true, $H_2$ false	$H_1$	8 of 100	$8/100=0.08$

Under the global null, the FWER is $0.02$, which is below $0.05$. That establishes weak control for the displayed global-null configuration. When $H_2$ is false and $H_1$ remains true, the FWER is $0.08$, which exceeds $0.05$. Thus the procedure does not have strong control. The example illustrates a logical point: one global-null probability cannot certify a guarantee over all configurations. It does not assert that every selection procedure has this error profile.

Account for Selection in the Error Event

When selection occurs, a false rejection of $H_i$ requires both that the arm can be tested and that the testing rule rejects its true null. It is useful to write this event explicitly. For a rule that tests only retained arms, let $C_i$ be the event that the arm crosses its rejection boundary. Then $R_i=S_i\cap C_i$, and for a true $H_i$,

P_\theta(R_i)=P_\theta(S_i\cap C_i) =P_\theta(S_i)\,P_\theta(C_i\mid S_i).

This identity follows from the definition of conditional probability when $P_\theta(S_i)>0$. It shows why a rejection probability conditional on selection is not, on its own, the FWER contribution of that arm. The selection probability also matters. Conversely, a small probability of selection does not guarantee strong control: the chance of rejection conditional on selection might be large, and the familywise probability must include all true arms.

Selection is generally based on interim outcomes, and later testing may use data related to those outcomes. Therefore the selection event and the final rejection event need not be independent. The product above is an identity using a conditional probability, not an assumption of independence. FWER calculations must use the joint trial process under each configuration being assessed. This is the key operational interpretation of the error events in sequential multi-arm testing.

Worked Example: Combining Selection and Conditional Rejection Probabilities

Suppose that, under a particular configuration where $H_1$ is true, arm 1 is selected with probability $0.20$. Among trials in which it is selected, the probability that the testing rule rejects $H_1$ is $0.20$. For this arm, the probability of a false rejection is $$ P(R_1)=P(S_1)P(C_1\mid S_1) =0.20\times0.20=0.040. $$

This is a 4% false-rejection probability for $H_1$ in this configuration. It is not necessarily the familywise error rate: if another true null can also be rejected, the FWER is the probability of the union of those false-rejection events, not just $P(R_1)$.

Now suppose a different configuration gives $P(S_1)=0.80$ and $P(C_1\mid S_1)=0.10$. Then $$ P(R_1)=0.80\times0.10=0.080. $$ Although the conditional rejection probability is lower in this second case, the unconditional false-rejection probability for arm 1 is higher. This calculation is why selection rates and conditional test behavior should be examined together, and under the configurations relevant to the control claim.

A Sufficient Route to Strong Control

One general way to establish strong FWER control is to bound each true-null rejection probability in every configuration and then use the union bound. This argument does not require independence between arms, between stages, or between selection and testing. Its assumptions are that the family is fixed, the error event for each arm is defined as above, and the stated probability bounds hold under every configuration in which the corresponding null is true.

Result: If, for every parameter value $\theta$ and every $i\in I_0(\theta)$, $P_\theta(R_i)\leq a_i$, where the nonnegative bounds $a_i$ satisfy $\sum_{i=1}^{m}a_i\leq\alpha$, then the procedure has strong FWER control at level $\alpha$.

To justify the result, write the event of at least one false rejection as a union over the true nulls. The probability of a finite union is at most the sum of the probabilities of its component events. Therefore

\operatorname{FWER}(\theta) =P_\theta\left(\bigcup_{i\in I_0(\theta)}R_i\right) \leq \sum_{i\in I_0(\theta)}P_\theta(R_i) \leq \sum_{i\in I_0(\theta)}a_i \leq \sum_{i=1}^{m}a_i \leq\alpha.

The argument holds for each $\theta$, including partial-null configurations, so it establishes strong rather than merely weak control. It is a sufficient condition, not a necessary one: the union bound can be conservative when false-rejection events overlap. Any less conservative approach still needs a valid bound on the familywise probability under the configurations it claims to cover.

Worked Example: Verifying a Strong-Control Bound for Three Arms

Consider a hypothetical three-arm design with $\alpha=0.05$. Suppose a full-procedure analysis establishes that, for every parameter configuration in which arm $i$'s null is true, $P_\theta(R_i)\leq0.015$, for each $i=1,2,3$. The bound includes the chance the arm is selected; it is not merely a conditional rejection probability.

For any configuration, let $I_0(\theta)$ be its true-null set. There can be at most three true nulls, so $$ \operatorname{FWER}(\theta) \leq\sum_{i\in I_0(\theta)}P_\theta(R_i) \leq 3(0.015)=0.045\leq0.05. $$

If only two nulls are true, the corresponding upper bound is $2(0.015)=0.030$; if only one is true, it is $0.015$. The bound covers the global null as well as each partial-null configuration. Subject to the stated probability bounds being valid for all relevant configurations, this proves strong FWER control. A bound obtained only under the global null would not justify the same conclusion.

How to Assess an Error-Control Claim

The central diagnostic question is: under which configurations were the error probabilities evaluated? A global-null calculation, whether theoretical or simulated, addresses weak control. To support a strong-control claim, the analysis must also cover partial-null configurations, or provide a valid argument that the global-null calculation bounds the error in those configurations. That latter implication cannot simply be assumed from the definition of FWER.

For a computational assessment, specify a set of configurations, simulate the full data-generating process under each one, apply the actual interim selection and stopping rules, and record whether any true null is rejected. If there are $B$ simulated trials and $E$ trials with at least one false rejection, the estimated FWER is $E/B$. This estimate is subject to Monte Carlo variability; it is not an exact probability or, by itself, proof of control. The configurations should include the global null and scientifically relevant mixtures of true and false nulls. For a theoretical assessment, the corresponding probabilities or bounds must likewise incorporate the selection procedure.

Useful diagnostic checks include confirming that the tested family and rejection direction are prespecified, that the simulation uses the same selection rule and stage structure as the intended design, and that false rejections are counted only for true nulls. Report both the configuration and its FWER; a single rate without its configuration is hard to interpret. If a claimed bound is based on armwise probabilities, check that each bound applies to the true arms under the partial-null configurations too.

A common mistake is to argue, “The overall false-positive rate is at most 5% when every arm is ineffective, so it is at most 5% whatever the effects on the other arms.” The first clause establishes only a global-null result. The second does not follow: changing which arms are effective can change the distribution of interim selection and the set of later tests. The correction is to establish the bound in every configuration required for strong control, or to use a sufficient argument that explicitly covers those configurations.

What Strong Control Does—and Does Not—Say

Weak control can be a relevant property when the only error question of interest is the global null. Strong control is the broader guarantee when conclusions may concern individual arms and some treatment effects may be present. A familywise guarantee addresses the probability of at least one false rejection; it does not say that every arm has the same error probability, nor does it describe the probability of selecting a clinically useful arm.

The union-bound approach is transparent and can be conservative; direct familywise calculations can use information about dependence, but their validity must still be established across the claimed configurations. Global-null calibration is easier to state and assess, but alone supports only weak control. Neither label describes power, and neither replaces a clear definition of the family, the selection rule, and the rejection events.

The main limitation of an empirical strong-control assessment is that the space of configurations may be large and simulated error estimates have uncertainty. A theoretical guarantee can address a broad parameter space, but may require assumptions about the data-generating model. In either case, the conclusion must match what was actually shown: a global-null result is weak control; a bound covering every configuration is strong control.

Check Your Understanding

Answer each question, then compare with the solutions below.

A four-arm procedure has global-null FWER $0.03$. What control does this establish, and what does it not establish?
For one true null, $P(S_i)=0.25$ and $P(C_i\mid S_i)=0.12$. Calculate its false-rejection probability.
For two true nulls, suppose $P(R_1)=0.018$ and $P(R_2)=0.021$. Give a union-bound upper bound on the FWER for this configuration.
Explain why checking strong control under the global null and under only one partial-null configuration does not generally establish strong control.

Solutions. 1. The result establishes weak control at level $0.03$ for the global-null configuration, assuming the probability is valid for the stated procedure. It does not establish control when one or more nulls are false. 2. By the conditional-probability identity, $P(R_i)=0.25\times0.12=0.030$, or 3%. 3. The union bound gives $\operatorname{FWER}\leq0.018+0.021=0.039$, or 3.9%; the actual union probability may be lower. 4. Strong control requires the bound for every parameter configuration. A finite set of checked configurations does not generally cover unexamined configurations unless an additional valid argument shows that those checks bound them.

Configuration	True nulls	Outcomes with at least one true null rejected	FWER
Both nulls true	\(H_1,H_2\)	2 of 100	\(2/100=0.02\)
\(H_1\) true, \(H_2\) false	\(H_1\)	8 of 100	\(8/100=0.08\)