Alpha Accounting In The Endpoint-Look Plane

Why Account for Alpha in Two Dimensions?

A trial with several endpoints and interim analyses has two distinct sources of multiplicity. Testing more than one endpoint creates opportunities for a false positive across endpoints; looking at an endpoint repeatedly creates opportunities to reject its null hypothesis at least once over time. A useful accounting plan therefore has two coordinates: which endpoint is being tested and at which interim look.

The endpoint coordinate assigns a share of the trial’s type I error budget to each endpoint. The look coordinate describes how that endpoint’s assigned budget is used over time. These are related, but they are not interchangeable. In particular, an endpoint’s budget is not a fresh allowance at every look, and a cumulative spending increment is not automatically the nominal significance level of that look’s test.

Definition: Suppose a trial has \(J\) endpoints and \(K\) planned looks. An endpoint-level alpha budget \(a_j\) is the maximum type I error allowance assigned to endpoint \(j\) across its full sequence of looks. A look-specific spending increment \(\delta_{jk}\) is the increment in the endpoint’s cumulative type I error spending assigned by the plan to look \(k\). The cumulative spending through look \(k\) is \(A_j(k)=\sum_{\ell=1}^{k}\delta_{j\ell}\).

This tutorial focuses on fixed, nonnegative budgets assigned in advance. The accounting identities are not themselves a method for selecting endpoint weights or constructing multiplicity procedures. They make the intended allocation explicit and provide checks that a procedure’s calibrated tests respect that allocation.

The Endpoint-Look Accounting Table

For each endpoint \(j\), the row of an endpoint-look table records the amount of its budget assigned at each look. Under a simple incremental accounting plan, the row sum is the endpoint budget:

\sum_{k=1}^{K}\delta_{jk}=a_j.

If the endpoint budgets together do not exceed the trial-wide level \(\alpha\), then

\sum_{j=1}^{J}a_j\leq\alpha.

Together, these conditions imply that the sum of all entries in the table does not exceed \(\alpha\). This is a bookkeeping statement about the plan. To interpret the entries inferentially, it matters how the tests at successive looks are calibrated. In a spending-function approach, \(A_j(k)\) is a target for the cumulative probability of crossing the endpoint’s rejection boundary by look \(k\) under its null hypothesis. The increments describe increases in that cumulative target, not necessarily marginal p-value cutoffs.

The table is often easiest to read in two stages. First, check the endpoint budgets down the rows’ totals. Second, within each row, check how each endpoint’s allowance is distributed across its looks. A row with a larger early increment spends more of its assigned allowance early; a row with a zero increment at a look assigns no additional cumulative spending there. Neither fact alone specifies the numerical boundary: that requires a sequential test calibrated to the endpoint’s information pattern and the joint distribution of its repeated test statistics under the null.

Worked Example: Splitting the Trial Budget Across Endpoints

Consider a hypothetical trial with two endpoints and a two-sided familywise level of \(\alpha=0.05\). Suppose the prespecified endpoint budgets are \(a_1=0.03\) and \(a_2=0.02\). Endpoint 1 has three looks, and endpoint 2 has two.

The endpoint allocation uses the full trial budget because \(0.03+0.02=0.05\). Suppose endpoint 1 is assigned increments \(0.012\), \(0.009\), and \(0.009\), while endpoint 2 is assigned increments \(0.008\) and \(0.012\). Then:

A_1(1)=0.012,\quad A_1(2)=0.012+0.009=0.021,\quad A_1(3)=0.021+0.009=0.030,

A_2(1)=0.008,\quad A_2(2)=0.008+0.012=0.020.

The row totals are \(0.030\) and \(0.020\), and the total across all entries is \(0.050\). The table therefore reconciles with the assigned endpoint budgets and the trial-wide level. This does not mean that the test at endpoint 1’s first look has a nominal significance level of \(0.012\), or that each later test can be run at its row’s increment. It means that the sequential procedure for endpoint 1 must be calibrated to have the specified cumulative spending targets under its null.

Why the Accounting Controls Familywise Error

Let \(H_{0j}\) denote the null hypothesis for endpoint \(j\), and let \(R_j\) be the event that the sequential procedure rejects \(H_{0j}\) at any of its looks. Strong familywise error control means that, for every configuration of true and false endpoint null hypotheses, the probability of rejecting at least one true null is at most \(\alpha\).

A sufficient set of conditions is that (1) the endpoint budgets are nonnegative and sum to at most \(\alpha\), and (2) for every endpoint \(j\), its full sequential procedure has probability no greater than \(a_j\) of ever rejecting \(H_{0j}\) when that null is true. The latter condition must hold for the relevant null parameter values, not merely at a convenient alternative or at one interim look. Under these conditions, no assumption of independence between endpoints is needed.

Justification: Let \(T\) be the set of endpoints whose null hypotheses are true. The event of at least one false rejection is \(\bigcup_{j\in T}R_j\). The union bound gives \(P(\bigcup_{j\in T}R_j)\leq\sum_{j\in T}P(R_j)\). If each true endpoint’s sequential test has \(P(R_j)\leq a_j\), then \(P(\bigcup_{j\in T}R_j)\leq\sum_{j\in T}a_j\leq\sum_{j=1}^{J}a_j\leq\alpha\). This holds for any dependence among endpoint statistics.

The key condition is about the probability of a rejection at any look for a true endpoint null. A procedure that controls each look’s error separately at a specified level does not automatically satisfy that condition when its tests are repeated. The sequential calibration must account for the repeated looks, for example by specifying cumulative spending targets and choosing boundaries that attain them under the assumed null distribution of the sequential statistics.

Worked Example: Converting Cumulative Targets to Increments

Suppose a hypothetical endpoint has budget \(a_1=0.024\) and three planned looks. Its prespecified cumulative spending targets are \(A_1(1)=0.006\), \(A_1(2)=0.015\), and \(A_1(3)=0.024\). Find the increments and check the budget.

The first increment equals the first cumulative target: \(\delta_{11}=0.006\). Each later increment is the difference between successive cumulative targets:

\delta_{12}=A_1(2)-A_1(1)=0.015-0.006=0.009,

\delta_{13}=A_1(3)-A_1(2)=0.024-0.015=0.009.

The increments are nonnegative, and their sum is \(0.006+0.009+0.009=0.024\), which equals the endpoint budget. The cumulative spending reaches, but does not exceed, \(a_1\) at the final look. A sequential test still has to be calibrated so its probability of a first boundary crossing by each look matches the intended cumulative targets under the endpoint null. The arithmetic alone does not determine the boundaries.

Spending Increments Are Not Per-Look P-Value Cutoffs

A potential source of confusion is that both spending increments and per-look significance levels may be written as small numbers. They describe different quantities. The increment \(\delta_{jk}\) records additional cumulative error spending assigned at a look. In contrast, a nominal per-look cutoff is a threshold used to decide whether the current test statistic is significant. Repeated test statistics are generally dependent, and an early crossing removes paths from the set of paths still at risk of first crossing later. Therefore, a target increase in cumulative crossing probability does not generally equal a marginal tail probability cutoff for the current statistic.

Worked Example: Spotting an Invalid Interpretation of a Spending Row

In a hypothetical three-look design, endpoint 1 has budget \(0.03\) and cumulative targets \(0.012\), \(0.021\), and \(0.030\). A reviewer proposes testing at nominal levels \(0.012\), \(0.009\), and \(0.009\), claiming these are the spending increments and hence control the endpoint’s sequential type I error.

The arithmetic of the increments is correct: \(0.012\), \(0.021-0.012=0.009\), and \(0.030-0.021=0.009\). The conclusion does not follow. If those numbers are simply used as marginal significance cutoffs at the three looks, the probability of at least one rejection depends on the joint distribution of the repeated statistics; it is not established by adding the increments as though the cutoffs were probabilities of disjoint first-crossing events.

The correction is to use the cumulative targets to calibrate sequential boundaries under the assumed null distribution, so the probability of crossing by each look follows the planned spending schedule. Alternatively, one could use a conservative design that explicitly bounds the sum of marginal per-look error probabilities. Either approach requires an actual calibration argument; labeling nominal cutoffs as “spending” is not enough.

Auditing a Plan and Recognizing Failure Modes

A sound audit checks both the allocation and the procedure that implements it. The endpoint-look table can reveal arithmetic or design inconsistencies, but it cannot by itself verify that a boundary has the claimed operating characteristics.

Check endpoint budgets.
Confirm \(a_j\geq0\) for every endpoint and \(\sum_j a_j\leq\alpha\). State whether any part of the trial-wide budget is deliberately unassigned.

Check each look schedule.
Verify that cumulative targets are nondecreasing, do not exceed their endpoint budget, and finish at the intended final spending. Compute increments by differencing successive cumulative targets.

Check sequential calibration.
Confirm that the boundary calculation accounts for repeated looks and the endpoint’s information times and null distribution. The target is control over the entire sequence, not merely at each look considered in isolation.

Check the familywise argument.
Ensure that each true endpoint null has its assigned full-sequence error bound and that the endpoint budgets together do not exceed \(\alpha\). If relying on a less conservative joint calculation, its dependence and error-control assumptions must be stated and verified.

Common warning signs include cumulative targets that decrease, increments that are negative, a row total larger than its endpoint budget, or endpoint budgets that exceed the trial-wide level. A further warning sign is documentation that reports only a nominal level for each look but never states the probability of rejecting a true endpoint null over the entire sequence.

A direct alternative is to assign marginal error bounds to every endpoint-look test and use the union bound across all cells. This can provide a simple conservative guarantee if the marginal bounds are valid under their nulls and their total is at most \(\alpha\). It is different from endpoint budgeting followed by sequential calibration: the former charges each cell directly, while the latter gives each endpoint a full-sequence budget and spends that budget over time. A joint calibration can sometimes use dependence information more efficiently, but requires a justified joint model and is not validated merely by a table of allocations.

The accounting framework has limits. It does not choose scientifically appropriate endpoint priorities, specify a testing hierarchy, or calculate group sequential boundaries. It also does not ensure that the assumed information times, test-statistic distributions, or null error bounds are correct. Those are separate design and implementation checks. If the actual looks or analysis rules differ from the plan, the planned spending table may no longer describe the implemented procedure.

Common mistake: “The increments add to the endpoint alpha, so testing at each look using its increment automatically controls error.” The correction is that increments allocate cumulative crossing probability; they are not generally nominal per-look cutoffs. Control follows only when the sequential tests are calibrated to the targets, or when a separate valid bound—such as an explicit sum of marginal error bounds—is established.

Check Your Understanding

Answer each question using the distinction between an endpoint budget and spending over looks.

A trial has \(\alpha=0.05\) and three endpoint budgets \(0.02\), \(0.02\), and \(0.015\). Does the allocation fit within the trial-wide budget? What amount remains unassigned?
An endpoint has cumulative targets \(0.005\), \(0.014\), and \(0.020\). Find its three increments and verify the final budget.
Why does allocating increments whose sum is \(0.03\) not, by itself, justify using them as nominal cutoffs at successive looks?
Does the union-bound familywise argument require independent endpoint statistics? State the condition it does require.

Solutions. (1) The total is \(0.02+0.02+0.015=0.055\), which exceeds \(0.05\) by \(0.005\); the allocation is invalid as stated, and no amount remains unassigned. (2) The increments are \(0.005\), \(0.014-0.005=0.009\), and \(0.020-0.014=0.006\). Their sum is \(0.020\), the endpoint budget. (3) The increments describe changes in cumulative crossing-probability targets. The probability of crossing at least once depends on the joint distribution across looks, so the increments are not generally marginal p-value cutoffs. The sequential boundaries need calibration, or a separate valid error bound must be shown. (4) No independence is required for the union bound. The required condition is that each true endpoint null’s full sequential procedure has rejection probability at most its assigned budget, with endpoint budgets summing to at most \(\alpha\).