Strong Control Under Arbitrary Dependence

Why Arbitrary Dependence Matters

In a multi-arm trial, treatment statistics may be dependent because arms share a control group, use common interim data, or are affected by the same trial-level conditions. Their dependence may be complex, and a model for it may be uncertain. A procedure that promises familywise error control only when the dependence is correctly specified can therefore be fragile.

A robust alternative is to use Bonferroni local tests inside a closed testing procedure. The key is to begin with p-values that are valid for the whole sequential test of each arm, accounting for all planned looks. Bonferroni’s inequality then controls every intersection of hypotheses without requiring any assumptions about how the armwise p-values depend on one another.

Definition: Let \(H_1,\ldots,H_K\) be the elementary treatment hypotheses in a prespecified family. The familywise error rate (FWER) is the probability of rejecting at least one true hypothesis. Strong FWER control at level \(\alpha\) means that this probability is at most \(\alpha\) for every configuration of true and false hypotheses in the family.

This is stronger than controlling the chance of any rejection when all hypotheses are true. Strong control must also hold, for example, when one arm’s null is true and the others’ nulls are false. A sequential procedure adds a further requirement: a p-value for an arm must remain valid after accounting for the repeated opportunities to cross a boundary at interim and final looks.

Armwise Sequential P-Values and Intersection Tests

For arm \(i\), let \(p_i\) denote a sequential p-value for testing \(H_i\). Its validity means that, whenever \(H_i\) is true, for every \(u\) between 0 and 1,

P(p_i \le u \mid H_i\text{ is true}) \le u.

This condition allows conservative or discrete p-values; it does not require a continuous uniform distribution. In a group-sequential design, \(p_i\) must account for the possibility of rejection at any permitted look, not just the look at which the analysis happens to stop. If arm selection or adaptation affects the testing rule, the p-value must also remain valid under that rule. A raw p-value calculated at a selected look does not automatically meet this condition.

For any nonempty subset \(I\) of the arms, define the intersection hypothesis \(H_I=\bigcap_{i\in I}H_i\). Under \(H_I\), every \(H_i\) for \(i\in I\) is true. The Bonferroni local p-value for this intersection is

p_I^{\mathrm{Bonf}}=\min\left(1,\ |I|\min_{i\in I}p_i\right).

The local test rejects \(H_I\) at level \(\alpha\) if \(p_I^{\mathrm{Bonf}}\le\alpha\), equivalently if at least one \(p_i\) in the subset is no greater than \(\alpha/|I|\). This test is valid under \(H_I\) regardless of dependence among the p-values.

Result: If each armwise sequential p-value is valid under its own null, then for any nonempty subset \(I\), the Bonferroni local test has type I error at most \(\alpha\) under \(H_I\), under arbitrary dependence between arms and across their sequential data.

The justification is a union bound. Under \(H_I\), an erroneous local rejection requires \(p_i\le\alpha/|I|\) for at least one \(i\in I\). Therefore,

P(\text{reject }H_I\mid H_I\text{ true}) \le \sum_{i\in I}P(p_i\le\alpha/|I|\mid H_i\text{ true}) \le |I|\frac{\alpha}{|I|} =\alpha.

No joint distribution appears in this argument. Dependence can make the actual probability smaller or larger than it would be under independence, but it cannot invalidate the upper bound.

Closing the Tests Across All Subsets

Testing only the full intersection \(H_{\{1,\ldots,K\}}\) is not enough to decide which individual arms can be declared effective. Closed testing supplies the rule: reject an elementary hypothesis \(H_i\) only if every intersection hypothesis containing \(H_i\) is rejected by its local test. In particular, this requires rejection of \(H_i\) itself and of the full-family intersection.

The closed adjusted p-value for \(H_i\) is the largest local p-value among all intersections containing it:

p_i^{\mathrm{closed}} = \max_{I:\,i\in I}p_I^{\mathrm{Bonf}}.

Thus \(H_i\) is rejected at level \(\alpha\) exactly when \(p_i^{\mathrm{closed}}\le\alpha\). Taking a maximum enforces all the intersection requirements at once. A small armwise p-value alone is not sufficient: an intersection containing that arm might still fail its local test.

Worked Example: Computing Closed Adjusted P-Values

Suppose a hypothetical three-arm trial has valid sequential p-values \(p_A=0.008\), \(p_B=0.021\), and \(p_C=0.120\). These values already account for all planned looks. Compute the Bonferroni local and closed adjusted p-values.

For the singleton intersections, the local p-values are 0.008, 0.021, and 0.120. For the two-arm intersections:

p_{AB}^{\mathrm{Bonf}}=2(0.008)=0.016,\quad p_{AC}^{\mathrm{Bonf}}=2(0.008)=0.016,\quad p_{BC}^{\mathrm{Bonf}}=2(0.021)=0.042.

For the full intersection, \(p_{ABC}^{\mathrm{Bonf}}=3(0.008)=0.024\). Each value is below 1, so truncation at 1 is unnecessary here. Taking the largest local value among intersections containing each arm gives

p_A^{\mathrm{closed}}=\max(0.008,0.016,0.016,0.024)=0.024,

p_B^{\mathrm{closed}}=\max(0.021,0.016,0.042,0.024)=0.042,\qquad p_C^{\mathrm{closed}}=\max(0.120,0.016,0.042,0.024)=0.120.

At \(\alpha=0.025\), arm A is rejected because \(0.024\le0.025\); arms B and C are not. The value 0.024 is the multiplicity-adjusted p-value for A under this closed procedure, not a probability that its null hypothesis is true.

Why the Closed Procedure Has Strong Control

The strong-control argument focuses on the set of hypotheses that are actually true. Let \(I_0\) be that set. If it is empty, no false rejection is possible. If it is nonempty and any true elementary hypothesis is rejected, the closed rule requires rejection of every intersection containing it, including \(H_{I_0}\). But \(H_{I_0}\) is itself true.

Consequently, the event of rejecting at least one true elementary hypothesis is contained in the event of rejecting the true intersection \(H_{I_0}\). The Bonferroni local test for that intersection has type I error at most \(\alpha\), by the union-bound argument above. Hence

P(\text{reject at least one true }H_i) \le P(\text{reject }H_{I_0}\mid H_{I_0}\text{ true}) \le \alpha.

This argument works for every possible true subset \(I_0\). It does not require that all nulls be true, and it does not rely on independence between arms. The sequential aspect is handled by requiring that each \(p_i\) already controls its own repeated-look testing; the closure then addresses multiplicity across arms.

Worked Example: Applying the Rule to Sequential Results

In a hypothetical four-arm trial, a prespecified boundary procedure has produced sequential p-values \(p_A=0.009\), \(p_B=0.014\), \(p_C=0.030\), and \(p_D=0.200\). Assume each value accounts for all interim and final looks and is valid under its arm’s null. At \(\alpha=0.05\), determine the closed decisions.

For arm A, the largest local p-value among the 8 intersections containing A is the full-family value \(4(0.009)=0.036\), so \(p_A^{\mathrm{closed}}=0.036\). For B, the largest is \(p_{BCD}^{\mathrm{Bonf}}=3(0.014)=0.042\), giving \(p_B^{\mathrm{closed}}=0.042\). For C, the largest is \(p_{CD}^{\mathrm{Bonf}}=2(0.030)=0.060\), so \(p_C^{\mathrm{closed}}=0.060\). Since the singleton test for D has p-value 0.200, \(p_D^{\mathrm{closed}}=0.200\).

Thus A and B are rejected at 0.05, while C and D are not. The fact that A has the smallest sequential p-value does not mean that its closed adjusted p-value equals 0.009: the decision must also pass all larger intersections containing A.

Assumptions, Diagnostics, and Alternatives

The guarantee is simple, but its assumptions are substantive. First, the family of hypotheses must be defined: the closure here ranges over all nonempty subsets of that family. Second, each \(p_i\) must be valid under \(H_i\) for the actual sequential design, including its stopping and selection rules. Third, each local intersection test must be applied as specified, and an elementary hypothesis must be rejected only after every containing intersection has been rejected.

Practical checks should therefore focus on how the p-values were obtained and how the rule is implemented. Confirm that a reported sequential p-value accounts for all looks rather than only the observed look. Verify that every intersection containing a claimed arm rejection was evaluated, and that the maximum, not the minimum, of its local p-values was used for the closed adjusted value. If arms were selected adaptively, check that selection has been accommodated in the p-value construction or testing procedure.

A tempting alternative is to calibrate a joint maximum statistic using an assumed dependence structure, as in Dunnett-type or MaxT procedures. When that joint calibration is valid, it may be more powerful than Bonferroni because it can use information about dependence. The trade-off is that its guarantee depends on the joint model or calibration being appropriate. Bonferroni closure is typically more conservative, but its type I error bound survives arbitrary dependence. These are different choices: robustness to dependence is gained at a possible cost in power.

Common pitfall: Using a per-look p-value as though it were a sequential p-value can invalidate the entire argument. For example, even for one null, two independent tests each run at 0.025 have probability \(1-(1-0.025)^2=0.04938\) of at least one false rejection. A repeated-look p-value or boundary procedure must account for the looks before it is used in the armwise Bonferroni tests.

Another error is to apply Bonferroni only to the full family and then declare every arm with a small individual p-value significant. That tests the global intersection but does not enforce the smaller intersections needed to make arm-specific claims. The closed adjusted p-value addresses this by checking every subset containing the arm. The cost is computational: with \(K\) arms there are \(2^K-1\) nonempty intersections. This can become burdensome for a large family, although the conceptual rule remains clear.

Worked Example: Bounding Error Under Arbitrary Dependence

Suppose a hypothetical four-arm trial has exactly two true null hypotheses, \(H_B\) and \(H_D\), and consider their intersection. At \(\alpha=0.05\), the Bonferroni local test rejects this intersection if \(p_B\le0.025\) or \(p_D\le0.025\). Assume both sequential p-values are valid under their respective true nulls.

Regardless of how the two statistics are related, the chance of falsely rejecting this intersection is bounded by

P(p_B\le0.025\ \text{or}\ p_D\le0.025) \le P(p_B\le0.025)+P(p_D\le0.025) \le 0.025+0.025=0.050.

If the closed procedure falsely rejects either true elementary hypothesis, it must also reject their true intersection \(H_{\{B,D\}}\). Thus the chance of any false rejection is at most 0.050. No correlation value is needed. The actual error probability may be lower; the bound does not assert that it equals 0.050.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

For an intersection of three true hypotheses, what Bonferroni threshold is used for each armwise sequential p-value when \(\alpha=0.03\)?
Two valid p-values for a true intersection have probability 0.012 and 0.018 of falling below 0.02. What upper bound does the union bound give for the probability that at least one falls below 0.02?
Suppose an arm has a small individual sequential p-value, but one intersection containing that arm has a Bonferroni local p-value greater than \(\alpha\). Can the arm be rejected by closed testing?
Why does a p-value calculated only at the observed interim look not necessarily qualify for this procedure?

Solutions. (1) The threshold is \(\alpha/|I|=0.03/3=0.01\). (2) The union-bound upper bound is \(0.012+0.018=0.030\); the events may overlap, so this is a bound, not necessarily the exact probability. (3) No. Closed testing requires every intersection containing the arm to be rejected, and the stated intersection fails its local test. (4) The observed look was one of several opportunities to reject. Unless the p-value accounts for the full sequential stopping rule, its null rejection probability over all looks may exceed its nominal level.