Common Errors In Two-Sided Error Accounting

Why Two-Sided Error Accounting Needs Care

In a group sequential test, the type I error is the probability, under the null hypothesis, that the monitoring rule ever makes a rejection. With two-sided monitoring, rejection may occur in either direction and at any scheduled look. Accounting therefore requires keeping three things distinct: the total rejection probability, its allocation between directions, and the probability of crossing at each look. These quantities are related, but they are not interchangeable.

The central issue is dependence. Test statistics observed at successive looks are generally correlated, and the design stops after a boundary is crossed. Adding marginal probabilities for separate looks can count some paths more than once. Likewise, doubling a one-sided rate is justified only when the two directional probabilities are equal and the one-sided number is the correctly defined cumulative sequential error—not merely a nominal per-look level.

Define the Error Event Before Calculating It

Suppose a trial has \(K\) scheduled looks. Let \(Z_k\) be the standardized test statistic at look \(k\), oriented so that large positive values favor one direction. Under the null hypothesis \(H_0\), assume the canonical joint normal model

(Z_1,\ldots,Z_K)\sim N_K(0,\Sigma),\qquad \operatorname{Corr}(Z_i,Z_j)=\sqrt{\frac{t_i}{t_j}}\quad (i<j),

where \(t_k\) is the information fraction at look \(k\), with \(0<t_1<\cdots<t_K=1\). This model applies, for example, when the standardized statistics arise from accumulating independent increments under the null. The calculations below presume fixed information times, deterministic boundaries, and no additional rule that can independently cause rejection.

Definition: Let \(u_k\) and \(l_k\) be the upper and lower efficacy boundaries at look \(k\), with \(l_k<u_k\). The trial stops at the first look \(k\) for which \(Z_k\ge u_k\) or \(Z_k\le l_k\). The upper directional type I error, \(\alpha_U\), is the probability under \(H_0\) that the first stop is an upper-boundary rejection. The lower directional type I error, \(\alpha_L\), is the corresponding probability of a lower-boundary rejection. The overall two-sided type I error is \(\alpha_{\mathrm{2s}}=\alpha_U+\alpha_L\).

The final equality holds because a trial that stops at the first efficacy boundary cannot have both an upper and a lower first stop. It is important that \(\alpha_U\) and \(\alpha_L\) are cumulative probabilities over the entire stopping rule. A probability at one look, calculated without accounting for the other looks and the stopping rule, is not itself a directional sequential error rate.

Three Accounting Errors

1. Doubling the wrong one-sided quantity

If a one-sided design has cumulative upper-boundary type I error \(\alpha_U\), doubling it is valid under a symmetric null distribution and a mirror-image rule: the lower boundary at every look must be the negative of the upper boundary, and the null distribution of the statistics must be unchanged when all signs are reversed. Then the upper and lower first-stop probabilities are equal, so \(\alpha_{\mathrm{2s}}=2\alpha_U\).

This does not justify doubling a nominal per-look significance level, an unadjusted one-sided \(p\)-value threshold, or a directional error rate from a different boundary rule. Nor is doubling valid just because the test has two directions: unequal boundaries, unequal tail budgets, a shifted null coordinate, or a lack of sign symmetry can produce \(\alpha_U\ne\alpha_L\).

2. Adding dependent look-specific probabilities

Let \(A_k\) be the event that the statistic crosses a specified boundary at look \(k\), considered as if monitoring continued to that look. In general, the probability of at least one crossing is

P_0\left(\bigcup_{k=1}^K A_k\right),

not \(\sum_{k=1}^K P_0(A_k)\). The sum counts paths that cross at multiple looks more than once. For two looks, the exact identity is \(P_0(A_1\cup A_2)=P_0(A_1)+P_0(A_2)-P_0(A_1\cap A_2)\). Across many looks, the overlap structure becomes more involved.

There is a second, related distinction. For an actual stopping rule, define \(F_{U,k}\) as the event that the first efficacy stop is an upper rejection at look \(k\). These events are disjoint, so \(\alpha_U=\sum_k P_0(F_{U,k})\). The marginal events \(A_k\), by contrast, are not necessarily disjoint. Summing first-stop probabilities is correct; summing marginal crossing probabilities is not the same operation.

3. Confusing tail allocation with overall control

A two-sided design may allocate different portions of its total error budget to the two directions. For example, specifying \(\alpha_U=0.03\) and \(\alpha_L=0.02\) allocates a total of \(0.05\), provided those numbers are cumulative first-stop probabilities under \(H_0\). They are not instructions to spend \(0.03\) and \(0.02\) separately at every look. Per-look boundary thresholds must be calibrated so that the resulting probabilities over the entire monitoring rule meet the intended directional budgets.

Conversely, assigning the full overall budget to each direction does not preserve that overall budget. If the two directional first-stop probabilities are each \(0.05\), their sum is \(0.10\), not \(0.05\). A design can be asymmetric in its allocation while still controlling the overall error, but the sum of its directional cumulative errors must meet the specified overall level.

Worked Examples

Worked Example: When Doubling a One-Sided Sequential Error Is Valid

Hypothetical design. Consider a two-look design under the canonical null model. Suppose an exact calibration of its upper boundary gives first-stop probabilities of \(0.008\) at look 1 and \(0.017\) at look 2. The boundary rule is symmetric: the lower boundaries are the negatives of the upper boundaries at both looks.

The cumulative upper directional error is the sum of the disjoint upper first-stop events:

\alpha_U=0.008+0.017=0.025.

Because the null distribution is sign-symmetric and the lower rule mirrors the upper rule, the lower first-stop probabilities are also \(0.008\) and \(0.017\). Thus

\alpha_L=0.025,\qquad \alpha_{\mathrm{2s}}=\alpha_U+\alpha_L=0.025+0.025=0.050.

Here, doubling is correct because \(0.025\) is the calibrated cumulative sequential error in one direction and the complete rule is symmetric. It would not follow from merely setting a one-sided nominal level of \(0.025\) at each look.

Worked Example: Why Marginal Look Probabilities Cannot Simply Be Added

Hypothetical calculation. Consider two standardized statistics under \(H_0\), each with a standard normal marginal distribution, and suppose their correlation is \(0.5\). For illustration, consider an upper crossing at zero at either look. This is a probability example, not a practical efficacy boundary. Let \(A_1=\{Z_1>0\}\) and \(A_2=\{Z_2>0\}\).

Each marginal crossing probability is \(P_0(A_1)=P_0(A_2)=0.5\). For a standard bivariate normal pair with correlation \(\rho\), the probability that both components are positive is \(1/4+\arcsin(\rho)/(2\pi)\). Substituting \(\rho=0.5\) gives

P_0(A_1\cap A_2) =\frac14+\frac{\arcsin(0.5)}{2\pi} =\frac14+\frac{\pi/6}{2\pi} =\frac13.

Therefore the probability of at least one crossing is

P_0(A_1\cup A_2) =0.5+0.5-\frac13 =\frac23\approx0.6667,

not \(0.5+0.5=1\). With an actual first-crossing stopping rule, the first-look crossing probability is \(0.5\), and the probability of crossing at look 2 after not crossing at look 1 is \(0.5-1/3=1/6\). These disjoint first-stop probabilities sum to \(0.5+1/6=2/3\), matching the union probability. This example makes the distinction explicit: marginal look events overlap, while first-stop events do not.

Worked Example: Tail Allocation Is Not a Per-Look Level

Hypothetical design target. Suppose a two-sided sequential procedure is intended to have total type I error \(0.05\), allocated as \(\alpha_U=0.03\) and \(\alpha_L=0.02\). Assume the null model and the calibrated boundaries have been checked to satisfy the fixed-look canonical model stated above. If the calibrated first-stop probabilities are \(0.018\) and \(0.012\) for upper stops at the two looks, and \(0.011\) and \(0.009\) for lower stops, then

\alpha_U=0.018+0.012=0.030,\qquad \alpha_L=0.011+0.009=0.020,

and consequently

\alpha_{\mathrm{2s}}=0.030+0.020=0.050.

The arithmetic is valid because each number is a first-stop probability for its direction and look. It does not imply that \(0.03\) and \(0.02\) were used as independent nominal levels at each look. If instead the two directional cumulative errors were each \(0.05\), the overall error would be \(0.05+0.05=0.10\), exceeding the stated target.

A Short Justification and Practical Checks

The accounting rule follows directly from partitioning the rejection event. Let \(F_{U,k}\) and \(F_{L,k}\) denote first stops in the upper and lower directions at look \(k\). Under the stated stopping rule, no path can belong to two of these events: after its first boundary crossing, monitoring stops. Their union is exactly the event of rejecting \(H_0\). Hence

\begin{aligned} \alpha_{\mathrm{2s}} &=P_0\left(\bigcup_{k=1}^K(F_{U,k}\cup F_{L,k})\right)\\ &=\sum_{k=1}^K P_0(F_{U,k})+\sum_{k=1}^K P_0(F_{L,k})\\ &=\alpha_U+\alpha_L. \end{aligned}

The second equality is valid because the first-stop events are mutually exclusive. The corresponding events of crossing a boundary at a particular look, without restricting to a first stop, need not be mutually exclusive. That is why summing the first-stop probabilities is exact, while adding marginal look-specific crossing probabilities is generally not.

When reviewing an error calculation, check the following before accepting its arithmetic:

Null model: Does the assumed joint distribution of the statistics match the design, including the information times and null value?
Stopping rule: Are probabilities for first stops, or for marginal crossings that could occur after an earlier stop?
Direction: Are upper and lower probabilities reported separately, and do they sum to the overall rejection probability?
Symmetry: If a one-sided rate has been doubled, are both the null distribution and the boundary rule symmetric under sign reversal?
Budget: Are the stated tail allocations cumulative over all looks, rather than levels to be spent afresh at each look?

A common alternative when exact joint probabilities are unavailable is to use a union bound: the probability of a union is no greater than the sum of its event probabilities. This can provide a conservative bound even when the events are dependent, but it is not an exact calculation of the sequential error and may be loose. Another approach is to calibrate the full joint stopping rule under the null. Whatever approach is used, the event being bounded or calibrated must be stated clearly.

These checks also reveal limits of simple accounting. The symmetry argument for doubling fails for asymmetric boundaries or unequal directional budgets. The first-stop partition presumes a specified rule that stops at the first efficacy boundary; additional stopping or rejection rules must be included in the rejection event. Finally, matching a nominal tail allocation does not establish control if the assumed null distribution, information times, or monitoring schedule is wrong.

Key takeaway: Define the rejection event first. Add disjoint first-stop probabilities to obtain overall two-sided error; do not add overlapping marginal look probabilities as if they were disjoint. Double a one-sided cumulative sequential error only when sign symmetry makes the two directional errors equal.

Check Your Understanding

For each question, identify the event being counted before doing the arithmetic. Solutions are included below.

A symmetric sequential rule has an upper first-stop error of \(0.024\). Under what conditions can its overall two-sided error be reported as \(0.048\)?
Two look-specific crossing events each have probability \(0.04\), and their intersection has probability \(0.01\). What is the probability of at least one crossing?
A design has cumulative first-stop errors \(\alpha_U=0.027\) and \(\alpha_L=0.018\). What is the overall two-sided error, and does it meet a \(0.05\) target?
Why is a tail allocation of \(0.03\) upper and \(0.02\) lower not equivalent to using nominal levels \(0.03\) and \(0.02\) at every look?

Solutions. (1) Doubling is justified if the null distribution is sign-symmetric, the lower boundaries mirror the upper boundaries at all looks, and \(0.024\) is the cumulative upper first-stop probability for that rule. Then \(\alpha_{\mathrm{2s}}=2(0.024)=0.048\). (2) By inclusion-exclusion, \(P(A_1\cup A_2)=0.04+0.04-0.01=0.07\). (3) The first-stop events by direction are disjoint, so \(\alpha_{\mathrm{2s}}=0.027+0.018=0.045\), which is below \(0.05\). (4) Tail allocations specify cumulative directional error over the full monitoring rule. Per-look nominal levels do not account by themselves for dependence, overlap, or the stopping rule, so they do not determine those cumulative probabilities.