From Endpoint Budgets to Interim Decisions
A trial may monitor more than one endpoint and examine each endpoint’s results at several planned interim looks. Each additional opportunity to reject a null hypothesis can increase the chance of at least one false rejection. A transparent way to control that risk is to divide a prespecified familywise error rate across the endpoint-look tests, then use the resulting local levels to set boundaries.
This tutorial works through a conservative Bonferroni allocation for two one-sided endpoint hypotheses and three prespecified looks. Its advantage is that the error-rate guarantee does not require independence between endpoints or between the repeated statistics at different looks. Its cost is that it can be more conservative than a design whose boundaries are calibrated using the joint distribution of the sequential statistics.
The example uses a total one-sided familywise error rate of \(\alpha=0.05\), meaning that the probability of falsely rejecting at least one true endpoint null hypothesis is to be no greater than 0.05. The hypotheses, direction of benefit, testing times, endpoint budgets, and look weights must be specified before examining interim results. The endpoint budgets must sum to no more than the total error rate, and the weights across looks for each endpoint must sum to one.
For the normal-boundary calculations below, assume that whenever \(H_{0,e}\) is true, each planned \(Z_{e,k}\) has a standard normal null distribution (or, more generally, gives a valid one-sided p-value at its assigned look). The test statistic must be calculated according to the prespecified analysis, with the beneficial direction represented by larger Z values. No independence assumption between the six statistics is needed for the Bonferroni guarantee. If the statistics do not have valid null distributions—for example, because the planned analysis or testing times are changed in response to unplanned information—the stated guarantee need not hold.
Worked Allocation and Boundary Calculation
Allocate \(\alpha_A=0.03\) to endpoint A and \(\alpha_B=0.02\) to endpoint B. These budgets add to the overall level: \(0.03+0.02=0.05\). For both endpoints, choose prespecified look weights \(w_1=0.20\), \(w_2=0.30\), and \(w_3=0.50\). The weights sum to one, so each endpoint’s full budget is distributed across its three possible tests.
Set \(\alpha_{e,k}=\alpha_e w_k\). This allocation gives the final look a larger local level than either interim look, but it still reserves error-rate budget for earlier decisions. The boundary is then calculated separately for every endpoint and look.
| Endpoint | Look | Local alpha calculation | \(\alpha_{e,k}\) | Boundary \(c_{e,k}\) |
|---|---|---|---|---|
| A | 1 | \(0.03(0.20)\) | 0.006 | 2.512 |
| A | 2 | \(0.03(0.30)\) | 0.009 | 2.366 |
| A | 3 | \(0.03(0.50)\) | 0.015 | 2.170 |
| B | 1 | \(0.02(0.20)\) | 0.004 | 2.652 |
| B | 2 | \(0.02(0.30)\) | 0.006 | 2.512 |
| B | 3 | \(0.02(0.50)\) | 0.010 | 2.326 |
For example, at endpoint A’s second look, the local level is \(0.03(0.30)=0.009\). Its one-sided boundary is therefore
A statistic of 2.366 or larger crosses this boundary; a smaller statistic does not. The boundary is not the critical value for a single unadjusted test at level 0.05. It reflects the specific local error-rate budget assigned to this endpoint at this look.
Worked Example: Calculate the Six Local Levels
Suppose the design uses the endpoint budgets and look weights above. Calculate the local alpha allocations and verify that they exhaust, but do not exceed, the familywise budget.
For endpoint A, multiply its budget, 0.03, by each look weight: \(0.03(0.20)=0.006\), \(0.03(0.30)=0.009\), and \(0.03(0.50)=0.015\). The three allocations sum to \(0.006+0.009+0.015=0.030\).
For endpoint B, the calculations are \(0.02(0.20)=0.004\), \(0.02(0.30)=0.006\), and \(0.02(0.50)=0.010\). They sum to \(0.004+0.006+0.010=0.020\).
Across all six tests, the sum is \(0.030+0.020=0.050\). Thus the allocation spends the full planned one-sided familywise error rate exactly once across the possible endpoint-look tests. Using the normal quantile formula for each local level gives the six boundaries in the table.
Why the Allocation Controls the Familywise Error Rate
The guarantee follows from the union bound. A false rejection occurs if at least one true endpoint null hypothesis is rejected at one of its looks. For each true null, the probability of rejection at look \(k\) is no greater than its local alpha allocation \(\alpha_{e,k}\). The probability of the union of these events is no greater than the sum of their probabilities:
Here \(\mathcal{T}\) is the set of endpoint null hypotheses that are actually true. The first inequality does not require the rejection events to be independent. It may be strict, particularly when tests are correlated, which is one reason this method can be conservative. If only one endpoint null is true, the bound can be smaller than 0.05; if both are true, the total allocated budget is 0.05.
The allocations are fixed for all planned opportunities to reject. Rejecting an endpoint early does not automatically transfer unused future budget to the other endpoint. Any recycling or other reallocation needs a prespecified rule with a valid error-control justification; simply spending the same budget again would break the bound.
Applying the Rules to Interim Results
Worked Example: Decisions at Three Prespecified Looks
Consider a hypothetical trial with information times 0.50, 0.75, and 1.00 of the planned final information. Suppose the one-sided statistics are as follows: endpoint A has \(Z_{A,1}=2.30\) and \(Z_{A,2}=2.40\); endpoint B has \(Z_{B,1}=2.60\), \(Z_{B,2}=2.50\), and \(Z_{B,3}=2.40\). Assume each statistic was calculated by the prespecified method and the design permits an endpoint to stop being tested after its null is rejected.
Endpoint A, look 1. Compare \(2.30\) with the boundary \(2.512\). Since \(2.30<2.512\), do not reject \(H_{0,A}\) at this look. The one-sided null tail probability is approximately \(1-\Phi(2.30)=0.0107\), which is greater than the local level 0.006.
Endpoint A, look 2. Compare \(2.40\) with \(2.366\). Since \(2.40\geq2.366\), reject \(H_{0,A}\) at look 2. Its one-sided null tail probability is approximately \(1-\Phi(2.40)=0.0082\), which is no greater than the local level 0.009. If the protocol stops testing endpoint A after its rejection, no later test of \(H_{0,A}\) is needed.
Endpoint B, look 1. The statistic \(2.60\) is below the boundary \(2.652\), so do not reject \(H_{0,B}\). Its one-sided tail probability is approximately 0.0047, which is above the local level 0.004.
Endpoint B, look 2. The statistic \(2.50\) is below the boundary \(2.512\), so do not reject \(H_{0,B}\) at look 2. Its tail probability is approximately 0.0062, just above the local level 0.006.
Endpoint B, look 3. The statistic \(2.40\) exceeds the boundary \(2.326\), so reject \(H_{0,B}\) at the final look. Its tail probability is approximately 0.0082, below the local level 0.010. The decisions are endpoint-specific: endpoint A is rejected at look 2 and endpoint B at look 3. These decisions do not imply that either endpoint causes an effect on the other.
Worked Example: Compute a Boundary Statistic From an Estimate
Suppose a hypothetical endpoint A measures improvement in a score, in points, with larger values favoring the treatment. At the final look, the estimated treatment effect is 5.25 points with standard error 2.50 points. The null value is zero improvement, and the prespecified statistic is the estimate minus the null value, divided by its standard error.
The statistic is
The units cancel, so \(Z_{A,3}\) is dimensionless. The final boundary for A is 2.170. Since \(2.10<2.170\), do not reject \(H_{0,A}\). The one-sided normal tail probability is approximately \(1-\Phi(2.10)=0.0179\), greater than the local level 0.015. Although the corresponding two-sided normal p-value is about 0.0357, that is not the decision rule for this prespecified one-sided test. The endpoint does not cross its allocated boundary.
Checks, Alternatives, and Limitations
Before applying a design, check that the direction of every test is consistent, all three looks and their weights were fixed in advance, the endpoint budgets sum to no more than 0.05, and each statistic is valid under its null. At each look, compare the statistic with the boundary for that exact endpoint and look—not with another endpoint’s boundary or with 1.96. A table of local levels and boundaries is useful because errors often arise from copying one endpoint’s threshold to another.
This fixed allocation differs from a sequential design that calibrates boundaries using the joint distribution of the repeated statistics for each endpoint, such as a group-sequential alpha-spending approach. A joint calibration accounts directly for the correlation induced by repeated use of accumulating data and may use the available error rate more efficiently. It still must account for multiplicity across endpoints. Conversely, the Bonferroni allocation here is simple and remains valid under arbitrary dependence, provided every component test is valid and every local level is counted in the total budget.
The method has important limits. Its boundaries can be conservative, especially when endpoint and look statistics are strongly correlated. It also provides no automatic rule for changing endpoint priorities, adding a look, changing the direction of testing, or reallocating alpha after an endpoint rejection. Such changes require a separately justified design. Finally, a boundary crossing is a statistical rejection under the specified testing rule; clinical importance requires interpretation of the estimated effect and its units as well as the test decision.
Check Your Understanding
Use the allocation in this tutorial unless a question specifies otherwise. Each problem is followed by its solution.
- What local alpha is assigned to endpoint B at look 2, and what is its approximate one-sided Z boundary?
- If endpoint A has \(Z_{A,1}=2.55\), what is the decision at look 1?
- Why does the familywise error argument not require independence between the six test statistics?
- Suppose a team tests endpoint B at all three looks and also reuses endpoint A’s unspent look-3 allocation for an additional test, without changing any prespecified rule. Why is that not justified by this allocation?
Solutions. (1) The local level is \(0.02(0.30)=0.006\), and \(c_{B,2}=\Phi^{-1}(0.994)\approx2.512\). (2) Endpoint A’s look-1 boundary is 2.512; since \(2.55\geq2.512\), reject \(H_{0,A}\) at look 1. (3) The union bound says the probability of at least one false rejection is no greater than the sum of the individual false-rejection probabilities, regardless of their dependence. (4) The guarantee counts each planned local level once. Adding an unplanned testing opportunity or reusing an allocation can make the sum of the error budgets exceed 0.05; the original bound therefore does not establish control for that altered procedure.