A Numerical Two-Stage Error Calculation

Why calculate error at each interim outcome?

In a two-stage test, the final decision may depend on what was observed at the interim analysis. A rule might stop for some interim outcomes, continue for others, or use different final-stage rejection cutoffs depending on the interim result. The overall type I error is the probability of rejection under the null, averaged over all possible interim outcomes. But that average alone does not show how much error the original design leaves available after each particular interim result. That distinction matters when evaluating a proposed data-dependent change.

This tutorial uses a small, fully hypothetical binomial design so every probability can be calculated exactly. The counts are illustrative, not data from a clinical study. The key calculation is to find the conditional rejection probability for each interim outcome, then combine those probabilities using the null distribution of the interim statistic.

Definition: For an interim statistic \(X\), the conditional type I error at outcome \(x\) is \(P(\text{reject the null}\mid X=x,H_0)\), using the specified continuation and final-stage rule. The unconditional type I error is \(P(\text{reject the null}\mid H_0)\), averaged over the possible interim outcomes. Under a composite null, type I error is the largest rejection probability over parameter values in that null.

A two-stage binomial model

Suppose the endpoint is binary and the null hypothesis is \(H_0:p\leq 0.20\), against the one-sided alternative \(H_a:p>0.20\). Stage 1 observes \(n_1=4\) participants and records \(X\), the number with the endpoint. If the trial continues, stage 2 observes a separate \(n_2=6\) participants and records \(Y\), the number with the endpoint in that stage. Assume independent Bernoulli outcomes with a common probability \(p\) across participants, no overlap between stages, and no unplanned looks or changes to the rule.

Under the boundary null \(p=0.20\), \(X\sim\operatorname{Bin}(4,0.20)\) and \(Y\sim\operatorname{Bin}(6,0.20)\), independently. For this upper-tail rejection rule, the boundary is the least-favourable value in the composite null: the probability of meeting any fixed upper-tail count threshold is nondecreasing in \(p\). Thus, checking the rejection probability at \(p=0.20\) controls it for every \(p\leq0.20\). This conclusion depends on the stated binomial model and the upper-tail nature of the rule.

Consider the original design: stop without rejection if \(X=0\); otherwise continue, and reject \(H_0\) if \(Y\geq3\). The final decision uses the stage-2 count and the prespecified rule. The conditional error after an observed \(x\) is therefore zero at \(x=0\), and is the same stage-2 tail probability for \(x=1,2,3,4\).

Worked Example: The Original Conditional and Unconditional Errors

First calculate the conditional error when the trial continues. Under \(H_0:p=0.20\), the stage-2 probability of three or more endpoint events is

\begin{aligned} e(x) &=P(Y\geq3\mid H_0)\\ &=1-\{P(Y=0)+P(Y=1)+P(Y=2)\}\\ &=1-\left\{\binom{6}{0}(0.20)^0(0.80)^6 +\binom{6}{1}(0.20)^1(0.80)^5 +\binom{6}{2}(0.20)^2(0.80)^4\right\}\\ &=1-(0.262144+0.393216+0.245760)\\ &=0.098880. \end{aligned}

Thus \(e(0)=0\), because the trial stops without rejection at \(X=0\), and \(e(x)=0.098880\) for each \(x=1,2,3,4\). The conditional error of 0.098880 is greater than the target overall level \(\alpha=0.06\), but that fact alone does not imply that the design's unconditional type I error exceeds 0.06. An outcome-specific conditional probability and an overall probability are different quantities.

The continuation probability under the null is

\begin{aligned} P(X\geq1\mid H_0) &=1-P(X=0\mid H_0)\\ &=1-(0.80)^4\\ &=0.5904. \end{aligned}

By the law of total probability, the unconditional type I error is the probability of continuing multiplied by the rejection probability after continuation:

\begin{aligned} P(\text{reject}\mid H_0) &=\sum_{x=0}^{4}P(X=x\mid H_0)e(x)\\ &=0.5904(0.098880)\\ &=0.058378752. \end{aligned}

The original design therefore has type I error about 0.05838 at the boundary null, below \(\alpha=0.06\). Under the binomial and monotonicity assumptions above, its type I error is no larger at any \(p\leq0.20\).

Deriving the averaging rule

The unconditional calculation is not a special property of the binomial distribution. It follows by partitioning the rejection event according to the mutually exclusive interim outcomes. Let \(R\) denote rejection. Since \(X\) can take values \(0,\ldots,4\), the events \(X=x\) partition the sample space. Therefore,

\begin{aligned} P(R\mid H_0) &=\sum_{x=0}^{4}P(R\cap\{X=x\}\mid H_0)\\ &=\sum_{x=0}^{4}P(R\mid X=x,H_0)P(X=x\mid H_0)\\ &=\sum_{x=0}^{4}e(x)P(X=x\mid H_0). \end{aligned}

The second equality is the multiplication rule for conditional probability. Each outcome-specific error is weighted by how likely that interim outcome is under the null. A rare interim outcome can have a large conditional error and still contribute little to the overall error; a common outcome with a large conditional error can contribute substantially.

More generally, if a proposed design changes the final-stage rule after observing \(X=x\), write its conditional rejection probability as \(e_{\text{new}}(x)\). Its unconditional type I error is calculated using the same interim distribution, provided the stage-1 data and its distribution have not changed:

P_{\text{new}}(R\mid H_0) =\sum_{x=0}^{4}P(X=x\mid H_0)e_{\text{new}}(x).

Assessing a data-dependent continuation rule

Now consider a proposed change: continue for every \(X\geq1\), but use the more permissive final cutoff \(Y\geq2\) when \(X\geq3\). For \(X=1\) or \(2\), retain the original cutoff \(Y\geq3\); for \(X=0\), stop without rejection. This rule makes the final rejection criterion depend on the interim result. Its conditional errors must be recalculated rather than assumed to remain those of the original design.

Worked Example: Calculating Error Under the Proposed Change

For \(X=1\) or \(2\), the conditional error remains 0.098880. For \(X=3\) or \(4\), the new cutoff is two events, so

\begin{aligned} e_{\text{new}}(x) &=P(Y\geq2\mid H_0)\\ &=1-\{P(Y=0)+P(Y=1)\}\\ &=1-(0.262144+0.393216)\\ &=0.344640,\qquad x\in\{3,4\}. \end{aligned}

The interim probabilities needed for the average are

\begin{aligned} P(X=1)&=4(0.20)(0.80)^3=0.4096,\\ P(X=2)&=6(0.20)^2(0.80)^2=0.1536,\\ P(X\geq3)&=P(X=3)+P(X=4)=0.0256+0.0016=0.0272. \end{aligned}

Thus \(P(X=1\text{ or }2)=0.5632\), and the proposed rule has unconditional error

\begin{aligned} P_{\text{new}}(R\mid H_0) &=(0.5632)(0.098880)+(0.0272)(0.344640)\\ &=0.055689216+0.009374208\\ &=0.065063424. \end{aligned}

This is approximately 0.06506, which exceeds \(\alpha=0.06\). The proposed rule does not control the unconditional type I error at the stated target under this null model. It also increases conditional error at \(X=3,4\), from 0.098880 under the original rule to 0.344640.

Conditional-error safeguard: A proposed adaptation preserves the original design's conditional-error guarantee if, at every interim outcome with positive null probability, its conditional rejection probability satisfies \(e_{\text{new}}(x)\leq e_{\text{old}}(x)\). Averaging these inequalities with the same nonnegative weights \(P(X=x\mid H_0)\) gives \(P_{\text{new}}(R\mid H_0)\leq P_{\text{old}}(R\mid H_0)\). This is a sufficient, outcome-by-outcome safeguard; it is stronger than checking only one unconditional average.

The condition is sufficient, not necessary for a particular numerical design to have unconditional type I error below a chosen level. An adaptation could spend more error at one interim outcome and less at another, yet still have an acceptable overall error for a specified null distribution. But that single average does not establish that the change preserves the original conditional-error guarantee. When a procedure is required to remain valid conditional on the observed interim information, each relevant outcome must be checked.

Worked Example: Checking a More Restrictive Continuation Rule

As a contrasting modification, stop without rejection if \(X=0\) or \(X=1\), and continue with the original cutoff \(Y\geq3\) if \(X\geq2\). At \(X=1\), the new conditional error is zero rather than 0.098880. At \(X=2,3,4\), it remains 0.098880, and at \(X=0\) it remains zero. The conditional error is therefore no larger than the original at any interim outcome.

Under \(H_0\), \(P(X\geq2)=1-P(X=0)-P(X=1)=1-0.4096-0.4096=0.1808\). The new unconditional error is

\begin{aligned} P_{\text{new}}(R\mid H_0) &=(0.1808)(0.098880)\\ &=0.017877504. \end{aligned}

The modification reduces the unconditional error and satisfies the outcome-by-outcome conditional-error safeguard. It is more conservative than the original design under this null model; error control alone does not say whether its power or scientific operating characteristics are desirable.

Diagnostics, alternatives, and limitations

A reliable calculation starts by writing down the complete decision rule for every possible interim outcome. For this example, the support is \(X=0,1,2,3,4\), so there are only five cases to check. A table can help prevent an outcome from being omitted.

Interim outcome	Original action	Original conditional error	Proposed cutoff	Proposed conditional error
\(X=0\)	Stop; do not reject	0	No continuation	0
\(X=1,2\)	Continue; reject if \(Y\geq3\)	0.098880	Reject if \(Y\geq3\)	0.098880
\(X=3,4\)	Continue; reject if \(Y\geq3\)	0.098880	Reject if \(Y\geq2\)	0.344640

Useful checks include confirming that the interim probabilities sum to one, that each conditional error is between zero and one, and that the unconditional error equals the weighted sum over all interim outcomes. Also verify that the final cutoff is applied to the correct stage-specific count and that a stopped trial cannot later reject. If a stage statistic depends on shared participants, changing sample sizes, or a different data-generating model, the stated independence and binomial calculations may no longer apply; the joint distribution must then be modeled appropriately.

Exact enumeration is especially practical when the interim statistic has a small discrete support. For a larger support, the same sum can be computed by software, but the rule and probability model still need to be specified correctly. Simulation can approximate operating characteristics when exact enumeration is impractical, but simulation introduces Monte Carlo uncertainty and does not replace a correct model or a complete decision rule. A conditional-error comparison focuses on what is permitted after each interim outcome; direct unconditional calculation focuses on the overall error under a specified null distribution. They answer related but distinct questions.

Common mistakes

Treating one conditional error as the overall error. The value 0.098880 applies after a continuation outcome in the original design; the unconditional error is the weighted average, 0.05838.
Using an unweighted average across interim outcomes. Outcomes do not generally have equal null probabilities. Use \(P(X=x\mid H_0)\) as the weight for each \(e(x)\).
Assuming that retaining the stage-1 rule retains the error guarantee. A changed final cutoff changes the conditional rejection probability and must be recalculated.
Checking only whether the average is below \(\alpha\). That answers the unconditional question for the specified model. It does not establish that the original conditional-error guarantee is preserved at every interim outcome.
Ignoring assumptions about the null distribution. A calculation at \(p=0.20\) controls the composite null here because the specified upper-tail binomial rejection probability increases with \(p\). That justification must not be carried over automatically to a different model or decision rule.

Key takeaway: Enumerate the interim outcomes, calculate the conditional rejection probability at each one, and average using their null probabilities to obtain unconditional type I error. To establish that a proposed adaptation preserves the original conditional-error guarantee, compare the new and original conditional errors outcome by outcome.

Check Your Understanding

Use the binomial model and original design above unless a question states otherwise. The solutions follow the questions.

Why can the original design have conditional error 0.098880 after continuation while its unconditional error is below 0.06?
Under the original rule, calculate the contribution of the interim outcome \(X=1\) to the unconditional type I error.
If the final cutoff \(Y\geq3\) is used for every \(X\geq1\), but the trial stops without rejection at \(X=0\), what is the unconditional type I error?
State the conditional-error safeguard for a proposed rule in terms of \(e_{\text{new}}(x)\) and \(e_{\text{old}}(x)\). Why does it imply an unconditional comparison under the same interim distribution?

Solutions. (1) The conditional error is calculated after a particular interim outcome, whereas the unconditional error averages across all outcomes, including \(X=0\), where rejection is impossible. Here the average is \((0.5904)(0.098880)=0.058378752<0.06\). (2) Since \(P(X=1)=0.4096\), its contribution is \((0.4096)(0.098880)=0.040501248\). (3) This is the original rule, so the unconditional error is \(0.058378752\), approximately 0.05838. (4) Require \(e_{\text{new}}(x)\leq e_{\text{old}}(x)\) at each interim outcome with positive null probability. Multiplying each inequality by \(P(X=x\mid H_0)\geq0\) and summing gives \(P_{\text{new}}(R\mid H_0)\leq P_{\text{old}}(R\mid H_0)\).