Why Graphical Procedures Control Familywise Error

Why Alpha Transfers Need a Familywise Error Argument

A graph makes endpoint priorities visible: each endpoint has an initial share of the familywise error rate, and rejecting an endpoint can pass some of its share to others. But a plausible-looking allocation is not, by itself, a proof of error control. In particular, a procedure must control the probability of any false endpoint rejection when any configuration of endpoint null hypotheses is true. This is strong familywise error control.

The connection that justifies a valid graphical procedure is the closure principle. Rather than checking only separate endpoint tests, closure requires a valid test of every nonempty intersection of endpoint hypotheses. Graph transfers provide a structured way to construct those local tests, and the sequential graph procedure can implement their rejections without explicitly testing every intersection.

Definition: Let \(H_1,\ldots,H_m\) be endpoint null hypotheses. For each nonempty subset \(I\) of \(\{1,\ldots,m\}\), define the intersection hypothesis \(H_I=\bigcap_{i\in I}H_i\). A procedure has strong familywise error control at level \(\alpha\) if, for every configuration of true and false endpoint nulls, the probability of rejecting at least one true endpoint null is at most \(\alpha\).

Assume that endpoint \(i\) has a p-value \(p_i\) satisfying \(P(p_i\leq u)\leq u\) for every \(u\in[0,1]\) when \(H_i\) is true. For a group sequential analysis, this assumption applies to the p-value for the endpoint’s entire interim-monitoring procedure, not to an unadjusted p-value at an arbitrarily selected look. No independence among endpoints is required for the Bonferroni argument below.

A weighted graph has initial weights \(w_i\geq0\) whose sum is at most one, and transition weights \(g_{ij}\geq0\) describing the fraction of endpoint \(i\)’s weight sent to endpoint \(j\) after \(H_i\) is rejected. Typically \(g_{ii}=0\), and each row satisfies \(\sum_{j\ne i}g_{ij}\leq1\). A live endpoint \(i\) is tested at its current level \(\alpha w_i\). On rejection, its weight is removed and the specified fractions are transferred to other live endpoints; any fraction not transferred to a live endpoint is discarded.

From the Graph to Tests of Intersections

To connect the graph with closure, consider a fixed nonempty set \(I\) and its intersection \(H_I\). Imagine that all vertices outside \(I\) have been rejected, so their alpha weight is passed along graph edges until it reaches a vertex in \(I\). Stop each path when it first enters \(I\); any weight that is lost or remains trapped outside \(I\) is discarded. Let \(q_i(I)\) be the total weight arriving at \(i\in I\), including the initial weight at \(i\). These quantities depend on the fixed graph and the set \(I\), not on the observed p-values. They satisfy

q_i(I)\geq 0,\qquad \sum_{i\in I}q_i(I)\leq \sum_{j=1}^m w_j\leq 1.

The associated local test rejects \(H_I\) if at least one \(i\in I\) satisfies \(p_i\leq\alpha q_i(I)\). If \(H_I\) is true, every \(H_i\) for \(i\in I\) is true. The union bound and the validity of each endpoint p-value therefore give

P(\text{local test rejects }H_I) \leq \sum_{i\in I}P\{p_i\leq\alpha q_i(I)\} \leq \alpha\sum_{i\in I}q_i(I) \leq \alpha.

This is a level-\(\alpha\) test of the intersection. Dependence among the endpoint statistics does not invalidate the union bound. The key requirement is that the weights used by this local test are fixed by the graph and \(I\), rather than chosen after inspecting p-values.

Theorem: A closed testing procedure rejects an endpoint hypothesis \(H_i\) only if it rejects every intersection \(H_I\) with \(i\in I\), using a level-\(\alpha\) local test for each nonempty \(I\). It then controls the familywise error rate strongly at level \(\alpha\).

A graph-based sequence of rejections is a shortcut for these closed tests. If the graph rejects any endpoint in a fixed set \(I\), consider the first rejected endpoint that belongs to \(I\). Before that rejection, all previously rejected endpoints lie outside \(I\). Its current weight is no greater than the weight that could reach it when all vertices outside \(I\) are treated as rejected. Thus its p-value crosses the corresponding local threshold \(\alpha q_i(I)\), and the local test of \(H_I\) rejects. This holds for each \(I\) containing a graph-rejected endpoint, so the graph’s endpoint rejections satisfy the closure requirement.

Worked Example: Tracking Weights and Checking Closure

Worked Example: A Three-Endpoint Graph

Suppose a hypothetical group sequential study has three endpoint p-values that are valid for their complete monitoring procedures. Let \(\alpha=0.05\), initial weights be \(w_1=0.50\), \(w_2=0.30\), and \(w_3=0.20\), and let the transitions be \(g_{12}=0.60\), \(g_{13}=0.40\), \(g_{21}=0.25\), \(g_{23}=0.75\), \(g_{31}=0.60\), and \(g_{32}=0.40\). Each transition row sums to one. Suppose the p-values are \(p_1=0.020\), \(p_2=0.018\), and \(p_3=0.030\).

Initially the test levels are \(0.05(0.50)=0.0250\), \(0.05(0.30)=0.0150\), and \(0.05(0.20)=0.0100\). Since \(0.020\leq0.0250\), reject \(H_1\). Its weight is transferred to the remaining endpoints: endpoint 2 receives \(0.50(0.60)=0.30\), and endpoint 3 receives \(0.50(0.40)=0.20\). The new weights are \(w_2=0.60\) and \(w_3=0.40\), giving test levels \(0.0300\) and \(0.0200\).

Now \(p_2=0.018\leq0.0300\), so reject \(H_2\). Of its weight, \(0.60(0.75)=0.45\) goes to endpoint 3. The remaining \(0.60(0.25)=0.15\) is directed to endpoint 1, which is already rejected, and is discarded. Endpoint 3 therefore has weight \(0.40+0.45=0.85\) and test level \(0.05(0.85)=0.0425\). Since \(0.030\leq0.0425\), reject \(H_3\).

For a closure check, take \(I=\{1,2\}\). Endpoint 3 lies outside this intersection, so its initial weight \(0.20\) transfers \(0.20(0.60)=0.12\) to endpoint 1 and \(0.20(0.40)=0.08\) to endpoint 2. Thus \(q_1(I)=0.62\) and \(q_2(I)=0.38\), with sum 1. The local levels are \(0.0310\) and \(0.0190\); \(p_1=0.020\leq0.0310\), so the local test rejects \(H_{\{1,2\}}\). More generally, for any \(I\) containing endpoint 1, \(q_1(I)\) is at least its initial weight \(0.50\). Hence \(p_1=0.020\leq0.05(0.50)=0.0250\), ensuring that the local test rejects every such intersection. The later graph rejections are consistent with the same closure logic: the first rejected endpoint within each intersection crosses its local threshold.

Why the Closure Argument Gives Strong Control

The closure theorem’s proof is short but covers every configuration of true endpoint nulls. Let \(T\) be the set of indices for which \(H_i\) is true.

Handle the empty true set.
If \(T\) is empty, there are no true endpoint hypotheses, so the event of at least one false rejection is impossible and the familywise error rate is zero.

Use the intersection of all true hypotheses.
If \(T\) is nonempty, \(H_T=\bigcap_{i\in T}H_i\) is one of the defined intersection hypotheses and is true.

Contain the error event.
If the closed procedure falsely rejects any true \(H_i\), it must reject every intersection containing \(H_i\), including \(H_T\). Therefore, the event of at least one false rejection is contained in the event that the level-\(\alpha\) local test of the true \(H_T\) rejects.

Apply local error control.
The local test of \(H_T\) has rejection probability at most \(\alpha\) when \(H_T\) is true. The probability of any false endpoint rejection is consequently at most \(\alpha\).

This is strong control because \(T\) was arbitrary: it may be any nonempty subset of endpoints, and the empty case has been handled separately. The proof does not assume that every endpoint null is true, or that endpoint statistics are independent.

Worked Example: Comparing with Bonferroni and Holm

Worked Example: A Graph Can Reallocate Error to Prioritized Endpoints

Use the same hypothetical p-values, \(0.020\), \(0.018\), and \(0.030\), and familywise level \(0.05\). Ordinary Bonferroni tests each endpoint at \(0.05/3=0.01667\). None of the three p-values is at most \(0.01667\), so this procedure rejects no endpoint.

For Holm’s step-down procedure, order the p-values as \(0.018\), \(0.020\), and \(0.030\). The first comparison is \(0.018\leq0.05/3=0.01667\), which is false. Holm therefore stops and rejects none. In the graph from the previous example, the initial weight gives endpoint 1 a level of \(0.0250\); its rejection then increases the level for endpoint 2 to \(0.0300\), and endpoint 2’s rejection raises endpoint 3’s level to \(0.0425\). The graph rejects all three.

This difference is not evidence that Bonferroni or Holm is invalid, nor does it establish that the graph is preferable for every study. The graph encodes a particular allocation and transfer plan. Its validity comes from the intersection tests, not from the fact that it produced more rejections in this hypothetical calculation.

Assumptions, Diagnostics, and Failure Modes

The mathematical guarantee depends on several details that should be checked in an implementation:

Initial budget: verify that every initial weight is nonnegative and \(\sum_i w_i\leq1\). A smaller sum is allowed; unused error budget is not automatically available elsewhere.
Transitions: verify that every transition weight is nonnegative and \(\sum_{j\ne i}g_{ij}\leq1\). Transfers to rejected endpoints, and unallocated fractions, must not silently be reused.
Sequential validity: each endpoint p-value or local sequential test must account for interim monitoring under its null. A nominal single-look p-value selected after looking at multiple interim results need not satisfy \(P(p_i\leq u)\leq u\).
Implementation: confirm that a rejected endpoint is removed and that only the specified shares are passed to live endpoints. Recalculate thresholds from the updated weights, rather than carrying forward an obsolete level.
Intersection logic: the local weights for each intersection must be determined from the graph, must sum to at most one, and must dominate the graph weight available at the first rejection within that intersection.

For example, if initial weights sum to 1 but one transition row sums to 1.20, rejecting its endpoint can create more than the available error budget. The resulting weights need not sum to at most one, and the weighted Bonferroni bound for an intersection is no longer guaranteed to be at most \(\alpha\). A graph with invalid transitions cannot be justified by the closure proof above.

Worked Example: A Transition Row That Breaks the Budget

Suppose two endpoints have initial weights \(w_1=0.60\) and \(w_2=0.40\), with \(\alpha=0.05\). If rejecting endpoint 1 transfers \(g_{12}=1.20\) of its weight to endpoint 2, endpoint 2’s updated weight would be \(0.40+0.60(1.20)=1.12\), giving it a nominal level of \(0.05(1.12)=0.0560\). The transition row exceeds one, so this is not a valid alpha transfer under the stated graph rules. In particular, a single true endpoint 2 could be tested above level \(0.05\) after endpoint 1 is rejected. The calculation signals a broken budget; it does not establish an exact familywise error rate for this invalid procedure.

A common mistake is to argue that error control follows merely because the weights add to at most one after each observed rejection. That check is necessary, but it does not explain why data-dependent changes after earlier rejections preserve control. The correction is to connect the graph to fixed, level-\(\alpha\) tests of all intersections and then invoke closure. Another mistake is to treat interim looks as if they were extra endpoints and apply an ordinary endpoint-level graph to unadjusted look-specific p-values. The endpoint p-values used by the graph must already be valid for the monitoring procedure.

The graph is an implementation shortcut, not a replacement for the assumptions behind local tests. Bonferroni and Holm offer familiar alternatives, while a graph can encode planned priorities and recycling. The graph’s limitations include dependence on the chosen weights and transitions, possible loss of power from conservative local Bonferroni tests, and failure of the guarantee if sequential endpoint tests or transfer rules are not valid. Detailed calibration of group sequential tests at the graph’s local levels is a separate design task.

Key takeaway: Valid alpha transfers conserve or discard the error budget, yielding local weighted Bonferroni tests for endpoint intersections. The graph’s rejections imply the corresponding local rejections; closure then bounds the probability of any false endpoint rejection by \(\alpha\), for every configuration of true nulls.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

Why does the local test of \(H_I\) have rejection probability at most \(\alpha\) when \(H_I\) is true, even if the endpoint p-values are dependent?
If the true endpoint set \(T\) is nonempty, which intersection is used in the closure proof? What happens when \(T\) is empty?
For \(\alpha=0.04\), suppose an intersection test has weights \(q_1=0.40\) and \(q_2=0.35\), with p-values \(p_1=0.015\) and \(p_2=0.014\). Does its weighted Bonferroni local test reject?
Why is checking only that each endpoint’s nominal level is no greater than \(\alpha\) not a sufficient argument for a data-adaptive graph procedure?

Solutions. (1) The union bound gives \(P(\text{reject }H_I)\leq\sum_{i\in I}P\{p_i\leq\alpha q_i(I)\}\leq\alpha\sum_{i\in I}q_i(I)\leq\alpha\); independence is not needed. (2) For nonempty \(T\), use the true intersection \(H_T\). If \(T\) is empty, no true null can be falsely rejected, so the familywise error rate is zero. (3) The local levels are \(0.04(0.40)=0.016\) and \(0.04(0.35)=0.014\). Since \(0.015\leq0.016\), the local test rejects. (4) Later weights depend on earlier graph rejections. Strong control requires showing that this procedure corresponds to level-controlled tests of all intersections; a pointwise check of nominal levels does not supply that closure argument.