The Pitfall Of Spending The Same Alpha Twice

Why the Same Alpha Cannot Be Spent Twice

In a multi-endpoint trial with interim looks, a decision rule can test several hypotheses at several times. Each individual test may use a seemingly small significance level, yet the chance of making at least one false rejection can exceed the trial’s planned level if the same error budget is repeatedly treated as new. The danger is especially easy to miss when endpoint-specific testing is combined with sequential monitoring or when a graph reallocates alpha after a rejection.

The central distinction is between an overall alpha budget and a local testing level. A local level specifies how stringent a particular rejection opportunity is. It is not automatically a fresh allowance for every later look, nor is alpha arriving at a graph node necessarily additional to alpha already assigned there. The method must account for the probability of false rejection over the whole decision process.

Definition: The familywise error rate (FWER) is the probability of rejecting at least one true null hypothesis in a family. If $\mathcal{H}_0$ denotes the set of hypotheses that are true in a particular configuration, then $$ \mathrm{FWER}=P(\text{reject at least one }H_i\in\mathcal{H}_0). $$ Strong control at level $\alpha$ means this probability is at most $\alpha$ for every configuration of true and false hypotheses. Under the global null, every hypothesis is true, so the FWER is the probability of at least one rejection.

A sequential design adds another index: a hypothesis may be tested at multiple looks. Its planned type I error must be controlled across those looks, not merely at each look in isolation. In standard group sequential testing, boundaries or an alpha-spending rule are calibrated so that the probability of crossing a boundary at any permitted look is bounded by the assigned error budget. For multiple endpoints, this sequential control must also fit within the multiplicity procedure’s overall budget.

How Repeated Testing Inflates Error

The simplest calculation makes the issue visible. Suppose there are $K$ tests, each performed at local level $a$, and under the global null their rejection events are independent. Each test fails to reject with probability $1-a$. Therefore, the probability that all $K$ tests fail to reject is $(1-a)^K$, and the FWER is its complement:

\mathrm{FWER}=1-(1-a)^K.

This is an exact result under the stated independence assumption. Without independence, the exact FWER depends on the joint distribution of the test statistics, but the union bound still gives $\mathrm{FWER}\leq Ka$. Thus setting each of $K$ tests to level $\alpha$ does not generally guarantee FWER control at $\alpha$; it only gives the potentially much looser bound $K\alpha$.

Worked Example: Reusing One Endpoint’s Alpha at Two Looks

Consider a hypothetical single-endpoint design with an overall target $\alpha=0.05$. An invalid rule applies a level $0.05$ test at each of two looks, treating the same budget as available twice. To isolate the bookkeeping issue, assume the two null rejection events are independent and each has probability $0.05$. This independence is a simplifying assumption for this calculation; it is not generally appropriate for cumulative interim test statistics.

The probability of no false rejection at either look is $$ (1-0.05)^2=0.95^2=0.9025. $$ Hence $$ \mathrm{FWER}=1-0.95^2=1-0.9025=0.0975. $$

The chance of at least one false rejection is therefore $9.75\%$, almost twice the intended $5\%$. The error is not that the level $0.05$ test is invalid at an individual look. It is that the rule has not controlled the probability of crossing either look’s threshold over the complete design.

This derivation also clarifies why the calculation cannot simply be copied to every group sequential design. Statistics based on cumulative data at successive looks are usually correlated. With strong positive correlation, repeated opportunities may add less error than independent tests; with another joint dependence structure, the result differs again. But correlation is not a license to reuse alpha: it means that the joint crossing probability must be calculated or bounded under the actual design. Group sequential boundaries are chosen with that joint probability in view.

Endpoint Budgets and Graph Transitions

Multiplicity procedures often allocate a portion of the overall alpha to each endpoint. A sequential procedure then has to control the probability of a false rejection for that endpoint across its looks, within the amount allocated to it. If a local endpoint level is $a_i$, applying a level-$a_i$ test at every look is not the same as spending a total of $a_i$ over those looks. The latter requires boundaries or incremental spending calibrated so that the probability of crossing any of the endpoint’s boundaries is no greater than its assigned budget.

A graph-based procedure adds a second accounting task. Its nodes represent hypotheses, and its weights determine how the total alpha budget is assigned. When a hypothesis is rejected, a prespecified rule may transfer its allocated weight to other hypotheses. Valid transfer changes where the remaining budget can be used; it does not create new budget. In particular, an implementation must not retain an allocation at a recipient and then add the donor’s alpha as though that alpha had not already been counted elsewhere. Nor may it restart the recipient’s full local sequential budget at every look after a transition.

Worked Example: Four Endpoint-Look Tests

Suppose a hypothetical multiplicity plan assigns each of two endpoints a local budget of $0.025$, so the initial allocations sum to the trial target $0.05$. An invalid implementation uses a level $0.025$ test for each endpoint at each of two looks, without calibrating the repeated opportunities. For illustration only, assume all four null rejection events are independent and each has probability $0.025$.

The probability of no rejection is $$ (1-0.025)^4=0.975^4=0.903687890625. $$ Thus the FWER is $$ 1-0.975^4=0.096312109375\approx0.09631, $$ or about $9.631\%$. Each endpoint’s two-look chance of at least one false rejection is $1-0.975^2=0.049375$, which is nearly twice its assigned budget of $0.025$. The calculation shows why endpoint allocation alone does not solve the across-look problem: the sequential rule must control each endpoint’s cumulative crossing probability as well.

The independence assumption makes the displayed FWER exact for this example, but the four tests in a real trial may be correlated across endpoints and looks. The numerical value should not be used as an estimate for such a trial. The diagnostic conclusion is general: four local tests at $0.025$ cannot be called a $0.05$-FWER procedure merely because the two initial endpoint allocations sum to $0.05$.

A graph transition can also be mishandled by adding the whole trial alpha to a recipient that already has its own allocation. The next example isolates that error with a deliberately simple testing sequence. It is not a recommended graph procedure; its purpose is to show the consequence of double counting.

Worked Example: Duplicating Alpha at a Recipient Node

Suppose a hypothetical two-endpoint design starts with $0.025$ allocated to each endpoint, for a total of $0.05$. Endpoint 1 is false and is assumed to be rejected, so there is no false-rejection concern from that endpoint in this example. Endpoint 2 is true. An invalid implementation tests endpoint 2 once at level $0.025$, keeps that opportunity, and then—after the graph transition—gives it a second, independent test at level $0.05$, mistakenly treating the full trial alpha as newly available. Assume the two endpoint-2 rejection events are independent under its null.

The probability endpoint 2 escapes both tests is $$ (1-0.025)(1-0.05)=0.975\times0.95=0.92625. $$ Therefore, the probability of falsely rejecting the true endpoint-2 null is $$ 1-0.92625=0.07375. $$ This is $7.375\%$, exceeding the $5\%$ trial target. The problem is not that transferring alpha is inherently invalid. It is that the procedure gave endpoint 2 its original opportunity and then added a full $0.05$ opportunity without accounting for the alpha already assigned and used. A valid transition must follow its specified weights and sequential calibration, not this duplicated allocation.

Justifying the Budget Check

A useful general justification comes from considering rejection events. Let $A_j$ be the event of a false rejection at testing opportunity $j$, where opportunities can include different endpoints and looks. A familywise error occurs exactly when at least one such event occurs. By the union bound,

P\left(\bigcup_{j=1}^{K}A_j\right)\leq \sum_{j=1}^{K}P(A_j).

Consequently, if a procedure guarantees that the sum of the relevant false-rejection probability bounds is at most $\alpha$, it controls the FWER at $\alpha$, regardless of dependence. This is a conservative justification, not a claim that every valid procedure must use Bonferroni allocation. More efficient group sequential and graph-based methods use the joint structure or a formal allocation-and-transfer rule, but they still need a valid accounting argument. Repeating a local level without such an argument does not establish control.

For a sequential endpoint procedure, one possible accounting target is that the probability of crossing any of its boundaries under its null is no greater than its assigned endpoint budget. For a graph procedure, transferred weight must be applied according to the graph’s rule, while the procedure continues to maintain the planned overall error control. These statements concern the full rejection rule, including every endpoint, look, and transition—not just the first test or a single nominal p-value.

Diagnostics, Alternatives, and Limitations

A practical audit should reconstruct the complete rule rather than inspect a table of nominal levels in isolation. For every endpoint, record its initial allocation, its test boundary at each look, and any changes after a rejection. Then check whether the cumulative false-rejection probability is controlled and whether each graph transition removes or reallocates budget as specified. Joint simulation under relevant null configurations can reveal mistakes that are hidden by reviewing local levels separately; simulation is a diagnostic and must represent the actual timing, dependence, and decision rules.

Warning sign: the same local level appears at every look. This may be valid only if a joint sequential calibration supports it; otherwise, it is an unexamined reuse.
Warning sign: incoming alpha is added without reducing or accounting for an existing allocation. Check whether the transition duplicates weight or merely reallocates it.
Warning sign: “each test is at level $\alpha$” is offered as the entire error-control argument. Individual validity does not imply familywise control across multiple tests.
Warning sign: a simulation reports only power or average rejection rates. Under configurations where specified null hypotheses are true, it should also assess the probability of at least one false rejection.

The main alternatives are not interchangeable shortcuts. Bonferroni allocation divides a total error allowance among tests and controls FWER by the union bound, often conservatively. A group sequential design calibrates boundaries across looks, accounting for the dependence induced by accumulating data. A graph-based procedure specifies how alpha is allocated and transferred across hypotheses. Each approach can be appropriate, but its assumptions and complete decision rule must match the implementation. A nominal p-value below a local threshold does not by itself establish that the whole multi-endpoint, multi-look procedure controls FWER.

The independent-test examples above are deliberately transparent, but they are limited: actual interim statistics are commonly correlated, and graph-based rejection opportunities may be conditional on earlier outcomes. For a real design, calculate or simulate the joint behavior under the specified null configurations using the actual look schedule, endpoint dependence, boundaries, and transition rules. Do not substitute the independent formula for that analysis unless independence is justified.

Key takeaway: Alpha is a probability budget for the complete testing rule, not a coupon that renews at each interim look or graph transition. Reuse is acceptable only when the full procedure is calibrated to control FWER; otherwise, repeated opportunities can inflate the chance of at least one false rejection.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

Under the global null, three independent tests are each conducted at level $0.01$. What is the exact FWER?
Why is it incorrect to conclude that two level-$0.025$ tests across two looks control FWER at $0.025$ for one endpoint?
Does positive correlation between interim statistics make it valid to apply the same local alpha at every look without further analysis?
A graph transition sends alpha to a node that already has an allocation. What accounting issue must be checked?
What should a joint simulation assess when checking a multi-endpoint, multi-look procedure?

Solutions. (1) Under independence, $\mathrm{FWER}=1-(1-0.01)^3=1-0.99^3=1-0.970299=0.029701$, or about $2.970\%$. (2) A level $0.025$ at each look is a local level, not automatically a total two-look budget. Even if the two rejection events were independent, the FWER would be $1-0.975^2=0.049375$, not $0.025$. (3) No. Correlation changes the joint crossing probability, but the complete sequential rule still requires valid calibration. (4) Check that incoming alpha is transferred according to the rule rather than duplicated on top of retained allocation or a restarted sequential budget. (5) It should estimate the probability of at least one false rejection under relevant configurations of true and false hypotheses, using the actual dependence, look timing, boundaries, and graph transitions.