Worked Example Of Unequal Tail Error Budgets

Why Allocate the Two Directions Unequally?

A two-sided group-sequential design can stop early for evidence in either direction. But the consequences of the two conclusions need not be equally serious, and the design need not assign the same amount of null rejection probability to each. For example, a treatment may be expected to improve an outcome, while evidence that it worsens the outcome is also important and may warrant a different degree of protection against a false directional conclusion.

This tutorial focuses on the meaning and use of unequal directional null error budgets. The key is to define each budget using the probability of a first crossing in that direction under the global null. These are probabilities across the full sequential monitoring rule, not per-look significance levels. The earlier material on first-passage events in opposite tails and joint null error provides the relevant sequential framework.

Definitions and Error Accounting

Let \(Z_k\) be the standardized test statistic at the \(k\)th of \(K\) prespecified analyses. Assume the convention that larger \(Z_k\) values favor benefit and smaller values favor harm. Let \(u_k\) and \(l_k\) be the upper and lower efficacy boundaries at look \(k\), with \(l_k<u_k\). The design stops when a boundary is crossed, or continues if neither is crossed and further analyses remain.

The operating characteristics below refer to the global null \(H_0\), under which there is no effect in either direction. The joint distribution of the statistics across looks must be the design’s specified null distribution; in a standard group-sequential design, this incorporates the information times and dependence between looks. Boundaries and looks must be specified in advance, or the analysis must otherwise account for any changes to the monitoring rule.

Definition: The upper-direction null error budget is \(\alpha_+=P_0(\text{the design first crosses an upper boundary})\). The lower-direction null error budget is \(\alpha_-=P_0(\text{the design first crosses a lower boundary})\). Here \(P_0\) denotes probability under the global null. If either crossing leads to rejection of \(H_0\), the overall two-sided null rejection probability is \(\alpha_{\mathrm{total}}=\alpha_++\alpha_-\).

The sum follows because a design that stops at its first boundary crossing cannot first stop in both directions on the same trial. Thus, the upper- and lower-crossing events are disjoint. The budgets are probabilities of the complete sequential events: they already account for the sequence of looks and its dependence structure.

P_0(\text{reject }H_0) =P_0(\text{first upper crossing})+P_0(\text{first lower crossing}) =\alpha_++\alpha_-.

An unequal allocation is a choice of how to divide the allowed overall error probability, not a change to the meaning of a false rejection. Under the global null, \(\alpha_+\) is the probability of a false benefit-direction conclusion and \(\alpha_-\) is the probability of a false harm-direction conclusion. Neither budget describes the probability that a particular conclusion is correct after observing data.

Worked Examples

Worked Example: Allocating a 5% Null Error Budget

Suppose a hypothetical trial sets the overall two-sided null rejection probability to \(0.05\). The design team assigns \(0.015\) to an upper crossing, interpreted as a benefit-direction decision, and \(0.035\) to a lower crossing, interpreted as a harm-direction decision. Check the total and compare the allocations.

The total is \(0.015+0.035=0.050\), so the two directional budgets add to the stated overall level. The harm-direction budget is larger. Its ratio to the benefit-direction budget is \(0.035/0.015=35/15=7/3\approx2.333\).

Therefore, the unfavorable direction receives \(7/3\) (about \(2.33\)) times as much null error as the benefit direction. Equivalently, the benefit-direction allocation is \(0.015/0.050=0.30\), or 30% of the total; the harm-direction allocation is 70%. These calculations describe the allocation of null crossing probabilities, not the relative numerical sizes of the upper and lower boundaries.

The design statement is: under the global null and the specified monitoring rule, the probability of a false benefit-direction stop is \(0.015\), and the probability of a false harm-direction stop is \(0.035\). Their sum gives a \(0.05\) probability of rejection in either direction.

Worked Example: Translating a Lower-Boundary Crossing

Consider a hypothetical three-look monitoring rule with upper boundaries \(u=(3.20,2.70,2.30)\) and lower boundaries \(l=(-3.40,-3.00,-2.60)\). Suppose the observed statistics are \(Z=(0.60,-3.10,-3.25)\). Determine whether the design stops and state the direction of the decision. These boundary values illustrate the decision rule; their null crossing probabilities would need to be calibrated separately for any proposed error-budget targets.

At look 1, \(-3.40<0.60<3.20\), so there is no crossing and monitoring continues. At look 2, \(-3.10\leq -3.00\), so the lower boundary is crossed. The first crossing is therefore in the lower direction at look 2. The statistic at look 3 does not change the decision: the design has already stopped.

If smaller values represent worse outcomes, the direction-specific report is that the monitoring rule stopped for evidence in the harm direction at the second analysis. This does not mean that harm is certain, nor does the observed path alone provide the null error probability. The latter is a design property calculated under \(H_0\) across possible paths, not a probability inferred from this single path.

In a design calibrated to unequal budgets, the corresponding false harm-direction conclusion is controlled by \(\alpha_-\), while the false benefit-direction conclusion is controlled by \(\alpha_+\). The numerical boundaries in this example are not asserted to have the budgets in the previous example; the example isolates how to translate a crossing into a directional decision.

Worked Example: Auditing Directional Calibration by Look

Suppose a hypothetical calibration report gives the following probabilities under the global null. Each entry is the probability that the first crossing occurs at that look in the indicated direction. Check the directional totals and the overall error probability.

First-crossing look	Upper direction	Lower direction
1	0.002	0.006
2	0.004	0.011
3	0.009	0.018

Sum the upper-direction probabilities: \(0.002+0.004+0.009=0.015\). Sum the lower-direction probabilities: \(0.006+0.011+0.018=0.035\). Thus \(\alpha_+=0.015\) and \(\alpha_-=0.035\). Because the entries refer to disjoint first-crossing events, the total null rejection probability is \(0.015+0.035=0.050\).

This report is consistent with a \(0.05\) overall level and the unequal allocation used in the first example. The individual entries are not per-look testing levels that can be applied independently to new data. They partition the calibrated first-crossing probabilities by look and direction.

Why the Accounting Works

The central justification is a partition of the rejection event. Write \(U_k\) for the event that the first boundary crossing is upper and occurs at look \(k\); define \(L_k\) similarly for a first lower crossing. With a first-crossing stopping rule, no path belongs to two different first-crossing events. In particular, it cannot belong to both an upper and a lower event. Rejection in either direction is the union of all these events.

\{\text{reject }H_0\} =\left(\bigcup_{k=1}^{K}U_k\right)\cup \left(\bigcup_{k=1}^{K}L_k\right), \qquad \alpha_+=\sum_{k=1}^{K}P_0(U_k), \qquad \alpha_-=\sum_{k=1}^{K}P_0(L_k).

Since the events in that union are disjoint, probabilities add:

P_0(\text{reject }H_0) =\sum_{k=1}^{K}P_0(U_k)+\sum_{k=1}^{K}P_0(L_k) =\alpha_++\alpha_-.

This identity explains why unequal allocation is coherent, but it does not calculate either budget from the boundary values. That calculation depends on the joint null distribution across looks and on the full boundary sequence. For example, changing a boundary at one look can change the probability of surviving to later looks, and therefore alter subsequent first-crossing probabilities. Calibrating the two budgets requires evaluating the sequential rule, not simply adding nominal one-look tail areas.

Interpretation, Diagnostics, and Alternatives

A directional error budget gives a precise repeated-monitoring statement under the global null. It does not, by itself, specify power under a beneficial or harmful effect, the probability of a particular stopping time, or the clinical importance of the effect at a crossing. Those are distinct properties of the design. The choice of allocation should reflect the consequences of the two types of false conclusion and should be agreed before interpreting trial results.

A useful calibration check is to verify all three quantities: the upper first-crossing probability, the lower first-crossing probability, and their sum. Also check that the stated probabilities were computed under the intended null model, look schedule, and stopping rule. A sum that exceeds the target suggests an error in calibration or accounting; a sum below the target may be intentional, but should not be described as using the full target level.

The main alternative is a symmetric allocation, with \(\alpha_+=\alpha_-=\alpha_{\mathrm{total}}/2\). Symmetry is straightforward when false conclusions in both directions have comparable consequences. Unequal allocation is appropriate when the design rationale supports treating the directions differently. Neither allocation is inherently more valid: the overall error control and the directional interpretation must match the prespecified design.

Boundary shape and tail allocation are related through calibration but are not interchangeable. Unequal upper and lower boundary values do not, by themselves, establish particular directional error budgets. Conversely, a target allocation does not uniquely determine every boundary value. The full sequence must be calibrated under the assumed dependence across looks. A design may also use a different approach to control overall error, but any alternative must still state clearly what event its error probability refers to.

Common Pitfalls and Limitations

Calling the budgets per-look levels. The quantities \(\alpha_+\) and \(\alpha_-\) are probabilities of first crossing over the full monitoring process, not the levels to use independently at each analysis.
Ignoring dependence across looks. Summing marginal upper-tail probabilities at each look generally does not give the upper first-crossing probability. Crossing at one look changes which paths remain at risk of crossing later.
Misstating the allocation ratio. For \(0.035\) versus \(0.015\), the ratio is \(0.035/0.015=7/3\approx2.33\), not three. Always divide the stated budgets and check the arithmetic.
Confusing direction with certainty. A lower-boundary stop supports a harm-direction decision under the prespecified rule; it does not prove harm or give the probability that harm is true.
Changing the rule after seeing results. Adding looks, altering boundaries, or changing which crossing triggers a conclusion can change error probabilities. The original calibration does not automatically cover a modified rule.

Unequal allocation also has limitations. It is a design choice about error control, not a remedy for weak measurement, model misspecification, or poor operational conduct. Its interpretation depends on the specified null model and monitoring plan. If those assumptions are not credible, a numerically correct allocation may not describe the actual false-decision risk.

Key takeaway: Define each directional budget as a full-process first-crossing probability under the global null. For a first-crossing two-sided rule, the overall null rejection probability is their sum; state each directional conclusion and allocation explicitly.

Check Your Understanding

Answer each question using the first-crossing interpretation of directional budgets.

A design has \(\alpha_+=0.012\) and \(\alpha_-=0.028\). What is its overall null rejection probability, and how many times as large is the lower-direction budget?
Why can the upper and lower first-crossing probabilities be added?
A report gives upper first-crossing probabilities of \(0.003\), \(0.005\), and \(0.007\) across three looks, and lower probabilities of \(0.004\), \(0.009\), and \(0.012\). Calculate both directional budgets and the total.
At an interim analysis, the statistic crosses the lower boundary first. What direction of decision does this support, and what does it not establish?

Solutions. (1) The total is \(0.012+0.028=0.040\). The lower-direction budget is \(0.028/0.012=7/3\approx2.33\) times the upper-direction budget. (2) Under a first-crossing rule, a path cannot first cross both directions, so the two rejection events are disjoint and their probabilities add. (3) The upper budget is \(0.003+0.005+0.007=0.015\); the lower budget is \(0.004+0.009+0.012=0.025\); the total is \(0.040\). (4) It supports a harm-direction stop under the prespecified rule. It does not establish that harm is certain or provide the probability that the conclusion is correct.