Least-Favourable Configurations for Strong Error Control

Why the Global Null Is Not Automatically Enough

Calibrating a maximum-statistic boundary under the global null controls the probability of at least one rejection when every null hypothesis is true. Strong familywise error control asks for more: that probability must remain at or below the target level for every configuration of true and false null hypotheses. A false arm can affect which arms are retained, which boundary is used, or whether a true-null arm is tested at all. Thus, the global-null calculation does not by itself establish strong control.

The key question is whether making some arms genuinely effective can increase the chance of falsely rejecting any remaining true null. A least-favourable-configuration argument answers this question by comparing the error probability at a partial-null configuration with that at the global-null boundary. The comparison needs conditions on the test’s behaviour; it is not a consequence of the word “maximum” or of Gaussian test statistics alone.

Definition: Let \(H_1,\ldots,H_K\) be the arm-specific null hypotheses, and let \(I\) denote the set of indices whose null hypotheses are true at a parameter configuration. The familywise error rate (FWER) at that configuration is \(P(\text{reject at least one }H_i\text{ with }i\in I)\). Strong FWER control at level \(\alpha\) means this probability is at most \(\alpha\) for every configuration. A configuration is least favourable if it attains the largest FWER over the configurations being considered. The global-null boundary has every \(H_i\) true, with each effect at its null boundary.

For clarity, consider one-sided hypotheses \(H_i:\theta_i\leq 0\) against \(\theta_i>0\). Positive effects are in the direction of rejection. A true null may have \(\theta_i<0\) or \(\theta_i=0\); false nulls have \(\theta_i>0\). The arguments below concern fixed nuisance parameters and a joint shift model for the stagewise statistics: the joint distribution has a common error component and its mean shifts in the direction of the effects. In particular, the covariance structure is unchanged as the effects vary. These assumptions make it possible to compare configurations using the same underlying random errors.

A Sufficient Condition for a Least-Favourable Global Null

Write \(Z\) for the vector containing the relevant arm-by-stage statistics. For a configuration with true set \(I\), let \(E_I(Z)\) be the event that the procedure rejects at least one \(H_i\) for \(i\in I\). A useful sufficient condition is that this event is increasing in evidence for the true arms and nonincreasing in evidence for the false arms. Increasing means that raising a true-arm statistic while holding the others fixed cannot turn a false rejection into a non-rejection. Nonincreasing for false-arm evidence means that raising a false-arm statistic cannot make rejection of a true null more likely for the same data on the true arms. The latter property can arise, for example, when stronger interim evidence for a competing arm can only make a true-null arm less likely to be selected.

This is a condition on the whole testing and selection rule, not merely on the marginal distribution of each statistic. In a joint shift model, use the same random error vector for the partial-null and global-null configurations. At a partial-null configuration, true-arm means are no greater than their boundary values, while false-arm means are greater than their boundary values. The monotonicity conditions then compare the false-rejection event path by path.

Theorem: Suppose the test has a fixed joint error distribution as effects vary, with true-arm evidence no larger than at its null boundary and false-arm evidence no smaller than at its null boundary. Suppose also that the event of rejecting any true null is increasing in the true-arm evidence and nonincreasing in the false-arm evidence. Then the FWER at any partial-null configuration is no greater than the FWER at the global-null boundary. Consequently, calibration of the global-null FWER at level \(\alpha\) gives strong FWER control at that level.

A short coupling argument proves the result. Let \(Z^0\) be the statistics under the global-null boundary, and construct statistics at the partial-null configuration using the same random errors. Denote the resulting vector by \(Z^\theta\). For every true arm, its evidence in \(Z^\theta\) is no greater than its evidence in \(Z^0\); for every false arm, its evidence in \(Z^\theta\) is no smaller. By the stated monotonicity,

\mathbf{1}\{E_I(Z^\theta)\} \leq \mathbf{1}\{E_I(Z^0)\} \leq \mathbf{1}\{\text{at least one rejection under }Z^0\}.

Taking expectations preserves both inequalities. The first expected value is the partial-null FWER. The last is the global-null FWER. Therefore the partial-null FWER is bounded by the global-null FWER, which is at most \(\alpha\) if it was calibrated at that level. The same reasoning also shows why a true effect below its boundary does not make this upper bound worse: lowering true-arm evidence cannot enlarge an increasing false-rejection event.

Worked Example: Fixed Armwise Boundaries

Consider three hypothetical independent standard-normal arm statistics under the global-null boundary. Reject \(H_i\) if \(Z_i\geq c\). Choose \(c\) to make the global-null FWER \(0.05\). Under this fixed armwise rule, an arm’s rejection depends only on its own statistic, so evidence on false arms cannot change the rejection decision for a true arm.

Under the global null, the probability that no arm crosses the boundary is \(\Phi(c)^3\). Thus,

1-\Phi(c)^3=0.05, \qquad 1-\Phi(c)=1-0.95^{1/3}=0.01695, \qquad c=\Phi^{-1}(0.98305)\approx 2.120.

Now suppose \(H_1\) and \(H_2\) are true at their boundaries, while \(H_3\) is false with a positive effect. The FWER counts rejections of \(H_1\) or \(H_2\), not rejection of the false null \(H_3\). Independence gives

P(\text{false rejection}) =1-\Phi(c)^2 =1-(0.98305)^2 \approx 0.03362.

The partial-null FWER, about \(0.03362\), is below the global-null FWER of \(0.05\). The positive effect on arm 3 does not alter the distribution or rejection rule for the two true-null arms. This example illustrates strong control for this particular rule and model; it does not establish that every selected or adaptive maximum-statistic procedure has the same property.

What Interim Selection Changes

With selection, a true arm’s chance of being tested can depend on another arm’s interim evidence. That dependence may make the global-null boundary least favourable, but it has to be shown for the actual algorithm. A simple protective pattern occurs when only one arm is retained and stronger evidence on a competing, false arm can only reduce the chance that a true-null arm is retained. For fixed true-arm data, that change cannot create an additional false rejection of the true arm.

This comparison is about the event of falsely rejecting a true null, not about the probability of choosing the best arm or the power to detect an effect. The earlier material on arm selection and on error events in sequential multi-arm testing distinguishes those performance and error events. Here, the important point is that a least-favourable argument must track the false-rejection event under the full selection rule.

Worked Example: A Protective One-Arm Selection Rule

Consider a hypothetical two-arm procedure with an interim statistic for each arm. It retains the arm with the larger interim statistic, breaking ties in favour of arm 1, and rejects the retained arm’s null only if that arm’s final statistic is at least \(2.00\). Assume a joint shift model in which increasing an arm’s effect raises its interim statistic while the same underlying errors can be used for comparison across configurations. Suppose arm 1 is true and arm 2 is false.

For fixed interim and final data on arm 1, raising arm 2’s interim evidence cannot make arm 1 more likely to be retained. If arm 1 is retained and its final statistic is at least \(2.00\), it is falsely rejected. If increasing arm 2’s evidence makes arm 2 the retained arm, arm 1 is no longer rejected. Thus, for each coupled data path, increasing arm 2’s interim evidence cannot change a non-rejection of true \(H_1\) into a rejection of true \(H_1\).

For example, suppose the interim statistics are \(1.40\) for arm 1 and \(0.80\) for arm 2 before applying an effect shift to arm 2, and arm 1’s final statistic is \(2.30\). With a shift of \(0.20\), arm 2’s interim value is \(1.00\); arm 1 is retained and its null is rejected. With a shift of \(0.80\), arm 2’s interim value is \(1.60\); arm 2 is retained, so \(H_1\) is not rejected. This pathwise comparison supports the theorem’s monotonicity condition for this rule. The global-null boundary is therefore least favourable for the FWER, provided the stated shift and selection assumptions hold and the rule is calibrated at the global null.

When the Argument Fails

The monotonicity condition is substantive. A rule may use evidence on one arm to make the boundary for another arm more permissive. Then a positive effect on a false arm can increase the chance of rejecting a true null, so the global-null boundary need not maximize FWER. The following counterexample uses independent statistics to isolate the problem; independence alone does not repair a rule that changes its threshold in this way.

Worked Example: A False Arm Activates a Liberal Boundary

Suppose \(Z_1\) and \(Z_2\) are independent, with \(Z_1\sim N(0,1)\) when \(H_1\) is true and \(Z_2\sim N(\theta_2,1)\). The procedure never rejects \(H_2\). It rejects \(H_1\) if \(Z_1\geq 1.50\) when \(Z_2\geq 1.00\), but requires \(Z_1\geq 2.50\) when \(Z_2<1.00\). This rule violates the protective condition: stronger evidence on arm 2 makes rejection of \(H_1\) easier.

Under the global null, \(\theta_2=0\), so the FWER is

\begin{aligned} \mathrm{FWER}_{0} &=P(Z_2<1)P(Z_1\geq 2.50) +P(Z_2\geq 1)P(Z_1\geq 1.50)\\ &\approx (0.8413)(0.00621)+(0.1587)(0.06681)\\ &\approx 0.01583. \end{aligned}

Now let \(H_1\) remain true and set the false-arm effect to \(\theta_2=3\). Then \(P(Z_2\geq 1)=\Phi(2)\approx 0.9772\), giving

\begin{aligned} \mathrm{FWER}_{\theta_2=3} &\approx (0.02275)(0.00621)+(0.9772)(0.06681)\\ &\approx 0.06543. \end{aligned}

The partial-null FWER, about \(0.06543\), exceeds the global-null FWER, about \(0.01583\). The global null is not least favourable for this rule. The calculation does not claim that this is an appropriate trial procedure; it shows why one cannot infer strong control from a global-null calculation without checking how false-arm evidence affects true-null decisions.

Diagnostics, Alternatives, and Limits

Before relying on a global-null least-favourable argument, check the actual rejection algorithm rather than only its final boundary. In a multi-stage design, enumerate how arm retention, stopping, and stage-specific testing depend on the observed statistics. Ask whether, holding the true-arm evidence fixed, increasing evidence on any false arm can make a true-null rejection easier. Also check that the assumed joint error distribution and covariance remain valid across the effect configurations being compared. A selection rule that changes after an interim look, or a model whose joint distribution changes with effects, may invalidate a simple coupling argument.

Signs of a problem: a boundary becomes more liberal when a competing arm’s statistic is large; a true arm is more likely to be tested under a partial-null configuration; or simulation shows a larger FWER for some non-null effect pattern than under the global null.
What global-null calibration establishes: it controls FWER at the global-null boundary under the model used for calibration. It establishes strong control only when a valid comparison, such as the theorem above, extends that bound to every partial-null configuration.
Alternatives when monotonicity is unavailable: assess FWER over relevant subsets of true hypotheses and plausible non-null effect patterns, or use a procedure whose construction gives a separate strong-control guarantee. A single global-null simulation is not a substitute for these checks.
Limitations: the sufficient condition may be conservative or fail even when a procedure has strong control for other reasons. Conversely, a few numerical checks cannot prove a uniform bound over every possible effect configuration.

A common mistake is to reason, “The maximum statistic was calibrated to have global-null FWER \(\alpha\), so strong FWER is at most \(\alpha\).” That conclusion skips the needed comparison. The correction is to state why a partial-null configuration cannot raise the probability of a false rejection—using a proved monotonicity argument, a different strong-control result, or a direct analysis over configurations.

Key takeaway: The global-null boundary is least favourable only when the test’s structure and the assumed model justify that comparison. For a protective monotone rule, a common-error coupling proves the partial-null FWER is no larger than the global-null FWER; without such a condition, global-null calibration alone is not strong error control.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

What is the difference between weak and strong FWER control?
In the coupling argument, why do true-arm and false-arm evidence move in opposite directions when comparing a partial-null configuration with the global-null boundary?
For a fixed armwise rule in which each \(H_i\) is rejected using only its own statistic, why can a positive effect on a false arm not increase the FWER among true arms?
In the counterexample, what feature causes the global-null boundary not to be least favourable?

Solutions. (1) Weak control bounds FWER when all null hypotheses are true; strong control bounds it for every configuration of true and false nulls. (2) Under one-sided hypotheses, true effects are at or below zero, so their evidence is no greater than at the boundary; false effects are positive, so their evidence is greater than at the boundary. The common-error coupling makes this comparison pathwise. (3) The rejection events for true arms depend only on their own statistics, whose joint distribution is unchanged by shifts in other arms under the stated model. The false-arm shift therefore cannot change which true-arm statistics cross their boundaries. (4) Evidence on arm 2 activates a lower threshold for arm 1. This makes a false rejection of true \(H_1\) more likely in the partial-null configuration than under the global null.