Why the Boundary Must Match the Error Claim
In a multi-arm study, a boundary determines when evidence is strong enough to reject one or more arm-specific null hypotheses. The central question is not merely whether a statistic crosses a boundary, but which probability of a false rejection the boundary controls. A rule that limits error for one arm at a time does not automatically limit the chance of making at least one false rejection across several arms.
The distinction persists when there are repeated looks. For arm \(k\), a false rejection can occur if its statistic crosses at any permitted stage. A familywise error occurs if this happens for at least one arm whose null hypothesis is true. The joint and armwise approaches control this union event in different ways: one calibrates a boundary for the joint distribution, while the other assigns error budgets to individual arms.
For a multi-stage procedure, write \(Z_{kt}\) for the test statistic for arm \(k\) at look \(t\), and \(b_{kt}\) for its upper efficacy boundary. A simple crossing rule rejects \(H_k\) if \(Z_{kt}\ge b_{kt}\) at any permitted look. The event of rejecting that arm is therefore $$ C_k=\bigcup_{t=1}^{T}\{Z_{kt}\ge b_{kt}\}. $$ The same reasoning applies if the boundary is a function of the stage or if the set of active arms changes, provided the rule's actual rejection event is used in the error calculation.
Joint Boundaries and Armwise Boundaries
A joint boundary is calibrated using the distribution of a statistic that accounts for multiple arms together, commonly the maximum across arms. At a single look with a common boundary \(c\), the rule rejects arm \(k\) when \(Z_k\ge c\); the global-null calibration chooses \(c\) so that the probability that the maximum exceeds it is at most \(\alpha\). With repeated looks, the joint event must include the permitted arm-by-stage crossings, not just the final look or a single arm.
An armwise boundary is calibrated separately for each arm. If the entire crossing probability for a true arm is bounded by \(\alpha_k\), and the allocations satisfy \(\sum_{k=1}^K\alpha_k\le\alpha\), the union bound gives strong FWER control without requiring independence between arms:
The phrase “entire crossing probability” matters. If an arm has several looks, its \(\alpha_k\) must bound the probability of crossing at any of those looks, not the probability of crossing at one look considered in isolation. Allocations may differ by arm, but each arm's guarantee must hold under the relevant configurations of other arms and under the procedure actually used.
A joint boundary can be less conservative than a Bonferroni allocation because it uses dependence among statistics. Its strong-control claim, however, needs a justification beyond global-null calibration alone. One common justification is subset pivotality: the joint distribution of the statistics corresponding to any set of true nulls is the same as their distribution under the global null. Then the event that any true-null statistic crosses a common joint boundary is contained in the event that the maximum over all arms crosses it under the global null. Without an assumption of this kind, or another valid strong-control argument, a global-null calculation may only establish weak control.
Worked Examples
Worked Example: Independent Arms at One Look
Suppose three independent, one-sided \(Z\)-statistics are observed at one look. Under the global null, each has a standard normal distribution. First consider a procedure that uses an unadjusted one-sided significance level of \(0.02\) for each arm. Find its FWER under the global null, then compare it with a joint maximum boundary at level \(0.05\).
For each arm, the chance of a false rejection under its null is \(0.02\). Independence gives
Thus the unadjusted armwise rule exceeds the target \(0.05\). For a common joint boundary \(c\), independence implies \(P(\max_k Z_k<c)=\Phi(c)^3\). Calibrating the maximum to have upper-tail probability \(0.05\) gives
The probability of at least one rejection under the global null is then \(0.05\), up to rounding. This calculation relies on the stipulated independence and normal null distributions. It illustrates global-null calibration; a strong-control claim for other configurations also needs a valid argument, such as subset pivotality.
Worked Example: Unequal Armwise Error Allocations
At a single look, three arm statistics have standard normal distributions under their respective nulls. Assign one-sided error budgets \(\alpha_1=0.025\), \(\alpha_2=0.015\), and \(\alpha_3=0.010\). Suppose the observed statistics are \(Z_1=2.10\), \(Z_2=2.20\), and \(Z_3=2.30\). Which arms cross their armwise boundaries, and what error guarantee follows?
The allocations sum to the target:
The one-sided normal critical values are approximately \(1.960\), \(2.170\), and \(2.326\), respectively. Comparing the observed statistics with these boundaries, arm 1 crosses because \(2.10>1.960\); arm 2 crosses because \(2.20>2.170\); and arm 3 does not cross because \(2.30<2.326\).
The corresponding one-sided p-values are approximately \(0.01786\), \(0.01390\), and \(0.01072\). These are below the allocated levels for arms 1 and 2, but the third p-value is above its allocation of \(0.010\). If each armwise test has its stated type I error bound, the union bound ensures FWER no greater than \(0.050\), regardless of the dependence among the three statistics. The observed crossings alone do not determine whether the rejections are false; that depends on which null hypotheses are true.
Worked Example: When a Joint Global-Null Boundary Has Strong Control
Consider two one-sided statistics with the model \(Z_k=\theta_k+\varepsilon_k\), where the error vector has the same joint distribution for every value of \((\theta_1,\theta_2)\). Each statistic has a standard normal distribution when its own null hypothesis, \(\theta_k=0\), is true. Suppose a common maximum boundary \(c\) is calibrated so that \(P_{(0,0)}(\max(Z_1,Z_2)\ge c)=0.05\). Explain why this calibration controls FWER strongly in this model.
If both nulls are true, the calibration directly bounds the probability of any false rejection by \(0.05\). If only \(H_1\) is true, then the false-rejection event is \(\{Z_1\ge c\}\). Since this event is contained in \(\{\max(Z_1,Z_2)\ge c\}\), and the distribution of the true-null statistic \(Z_1\) is unchanged when \(\theta_2\) changes, its probability is no greater than the corresponding global-null maximum probability, \(0.05\). The same argument applies when only \(H_2\) is true.
The unchanged joint error distribution for the true-null statistics supplies the needed subset-pivotality condition here. If that condition does not hold, the containment of events alone is insufficient: the probability of the maximum event under the alternative configuration might differ from its global-null calibration. Strong control must then be established another way, for example through valid armwise allocations or intersection-specific testing.
Why the Joint Argument Works—and Where It Can Fail
The key containment argument is short but useful. Let \(C_k\) denote crossing for arm \(k\), and let \(C_{\mathrm{all}}=\bigcup_{k=1}^K C_k\) denote any crossing across all arms. Since \(I_0\) is a subset of all arms,
Under subset pivotality, the joint distribution needed to evaluate crossings among true-null arms is their global-null distribution. The true-null crossing event is also contained in the all-arm crossing event used to calibrate the joint boundary. Together these facts justify strong control. If the joint distribution of true-null statistics changes when other hypotheses are false, the global-null probability does not, by itself, bound the probability in the configuration being tested.
A joint calibration should use the dependence structure and rejection rule that will actually be used. For a repeated-look design, a maximum over arms at the last look is not automatically a calibration of the event “any arm crosses at any look.” The latter event includes more opportunities to reject. Likewise, a maximum calibrated for all originally planned arms is not automatically appropriate if the testing rule selects arms using data and then changes which boundaries apply.
List the arms, looks, and selection decisions that can lead to a rejection. Define \(C_k\) to include every permitted crossing for arm \(k\).
For joint calibration, state the joint-null distribution and the event whose probability is calibrated. For armwise calibration, state the per-arm crossing bound and the allocations.
Verify subset pivotality or another strong-control justification for the joint method. For the armwise method, verify each bound remains valid for every relevant configuration and that the allocations sum to at most \(\alpha\).
Only after the error guarantees are established should the boundaries be compared for stringency or power. A less stringent boundary is not preferable if its claimed error control does not hold.
Diagnostics, Alternatives, and Limitations
Warning signs of a boundary mismatch include calibrating each arm at level \(\alpha\) and then claiming familywise level \(\alpha\); calibrating a joint maximum under the global null but claiming strong control without checking the distribution under partial-null configurations; or calculating a per-look armwise error while ignoring crossings at other looks. Selection creates a further risk: a per-arm guarantee for a prespecified arm does not automatically apply after data-dependent selection. The selection rule and its effect on the rejection event must be included in the calibration or covered by a valid error allocation.
| Approach | How it controls error | Main advantage | Main condition or cost |
|---|---|---|---|
| Joint maximum boundary | Calibrates a joint crossing event using the arm-and-stage dependence structure | Can use dependence and avoid some conservatism | Needs a valid joint distribution and a strong-control justification |
| Armwise allocation | Bounds each arm's full crossing probability and sums allocated levels | Simple guarantee under arbitrary dependence | May be conservative; each arm's bound must cover all looks and relevant configurations |
| Intersection-specific testing | Requires tests for relevant intersections and combines their decisions under a closed-testing rule | Can provide strong control without relying only on a global-null test | Requires a family of valid local tests and can be more involved to implement |
The armwise union-bound method is robust to dependence, but it can sacrifice power because it does not exploit the joint distribution. Conversely, joint calibration can gain efficiency when the dependence is well characterized, but sensitivity to covariance assumptions is a limitation: a poorly estimated or incorrectly specified covariance can misstate the probability of joint crossing. When the number of arms, active set, or test-statistic dependence differs from what the calibration assumes, the nominal guarantee may no longer apply.
A frequent mistake is to reason, “each arm has a five-percent boundary, so the familywise error is five percent.” This confuses a marginal error guarantee with a bound on a union of events. With three independent tests each at level \(0.02\), for example, the familywise error is \(0.05881\), not \(0.02\). The correction is to calibrate the joint crossing event, or to allocate error across arms so the sum of valid per-arm bounds is no greater than the familywise target. Dependence does not rescue the general argument: it can change the union probability, but it does not make unadjusted armwise levels a universal FWER guarantee.
Check Your Understanding
Answer each question, then compare with the solutions.
- Two independent one-sided tests each use level \(0.025\). What is the global-null FWER?
- Why does the union bound give strong FWER control when each true arm has a valid crossing probability no greater than its allocated \(\alpha_k\), with \(\sum_k\alpha_k\le\alpha\)?
- What additional condition can allow a joint boundary calibrated under the global null to control FWER under partial-null configurations?
- For three looks at one arm, why is it not sufficient to assign level \(\alpha_k\) separately to each look and then claim the arm's total crossing probability is at most \(\alpha_k\)?
- Give one reason a joint boundary may be less conservative and one reason it may be less reliable than armwise allocation.
Solutions. (1) Independence gives \(1-(1-0.025)^2=1-0.975^2=0.049375\), approximately \(0.04938\). (2) The false-rejection event is a union over the true arms, so its probability is no greater than the sum of their crossing probabilities, which is bounded by \(\sum_{k\in I_0}\alpha_k\le\alpha\). This does not require independence. (3) Subset pivotality is one sufficient condition: the joint distribution of statistics for true nulls must be unchanged by which other hypotheses are false. (4) Crossings at different looks are additional chances for rejection; the union of look-specific events can have probability greater than the probability at any one look. The arm's full sequential crossing probability must be bounded by \(\alpha_k\). (5) A joint boundary can exploit dependence among arms and thereby avoid some conservatism. It can be less reliable if the assumed joint distribution or covariance is misspecified, or if global-null calibration is used without a strong-control justification.