Selecting From A Menu Of Continuation Designs

Why Allocate Conditional Error Across a Menu?

At an interim analysis, a trial team may face a small number of prespecified continuation designs. For example, operational constraints might determine whether the trial continues under option A or option B. The choice may depend on interim data or other information available at that time. The statistical requirement is not that the options be identical. It is that the selected option, at every possible interim state under the null hypothesis, spend no more type I error than the original design had available at that state.

The conditional error preservation result gives the relevant comparison: calculate the original design’s probability of eventual rejection conditional on the interim information, then require the continuation rule to respect that conditional budget. With a menu, this check must cover each option that could be selected, not merely the option expected to be selected or the average error across options.

Definition: Let $\mathcal F$ denote all information available when the continuation option is chosen, and let $R_0$ be the event that the original design rejects $H_0$. The original conditional error is $$ C(\mathcal F)=P_{H_0}(R_0\mid\mathcal F). $$ For a candidate continuation option $j$, let $R_j$ be its eventual rejection event. Its conditional rejection probability is $C_j(\mathcal F)=P_{H_0}(R_j\mid\mathcal F)$. A conditional error allocation assigns each candidate a nonnegative budget $b_j(\mathcal F)$. Option $j$ respects its allocation if $C_j(\mathcal F)\le b_j(\mathcal F)$ for every possible interim state under the null.

For a composite null hypothesis, the condition must hold under every null parameter value, or under the specified uniform conditional-error bound used for the trial. Writing $P_{H_0}$ below represents the applicable null model; it is not a license to check only a convenient null value when the actual guarantee is required over a composite null. The menu, its allocation rules, and the selection rule should be specified before the continuation outcomes are observed.

A Simple Planned Design and Its Conditional Error

Consider a one-sided normal example. Under $H_0$, suppose the interim statistic $Z_1$ and the future increment $Z_2$ are independent standard normal random variables. The original design rejects when their equally weighted standardized combination exceeds $c$:

\frac{Z_1+Z_2}{\sqrt{2}}>c,\qquad c=z_{1-\alpha}.

Here $z_{1-\alpha}$ is the standard normal quantile with upper-tail probability $\alpha$. The independence and standard normal assumptions are part of this illustrative model; in an actual design, the joint null distribution used to calculate conditional error must match the planned analysis. Given $Z_1=x$, the original rejection rule is equivalent to $Z_2>\sqrt{2}c-x$. Therefore its conditional error is

C(x)=1-\Phi(\sqrt{2}c-x),

where $\Phi$ is the standard normal cumulative distribution function. This is a function of the observed interim statistic, not a single fixed number. An interim result that is more favorable to rejection leaves more conditional error available; a less favorable result leaves less.

Worked Example: Calculating the Original Conditional Error

Suppose the planned one-sided type I error level is $\alpha=0.025$, so $c=z_{0.975}\approx1.9600$. At an interim value $Z_1=1.000$, calculate the original conditional error.

The continuation boundary for $Z_2$ is $$ \sqrt{2}c-Z_1=(1.4142)(1.9600)-1.000=1.7718. $$

Thus $$ C(1)=1-\Phi(1.7718)\approx0.0382. $$

Conditional on this interim result, the original design would reject with probability about 0.0382 under the null. This exceeds 0.025 without contradicting the original design’s 0.025 unconditional type I error: conditional error can be above or below the unconditional level for particular interim outcomes. The relevant preservation condition compares the continuation rule with $C(1)$ at this interim state.

Allocating a Budget Across Candidate Designs

A straightforward allocation assigns a share of the original conditional error to each candidate. Choose nonnegative, prespecified weights $w_j(\mathcal F)$ whose sum is at most one, and set

b_j(\mathcal F)=w_j(\mathcal F)C(\mathcal F), \qquad \sum_{j=1}^{J}w_j(\mathcal F)\le1.

Then require $C_j(\mathcal F)\le b_j(\mathcal F)$ for every option $j$ that may be selected and every possible interim state. This is a conservative shared allocation: even the sum of the option budgets cannot exceed the original conditional error. Since the trial selects only one option, the sum restriction is stronger than is strictly necessary for selection alone. A simpler sufficient condition is $C_j(\mathcal F)\le C(\mathcal F)$ for every candidate. The shared allocation is useful when the design team wants explicit, separate error budgets rather than giving each option the entire available conditional error.

For a continuation test based on a standard normal statistic $Z_2$, with rejection rule $Z_2>k_j(\mathcal F)$, its conditional rejection probability is $1-\Phi(k_j(\mathcal F))$, provided the future statistic has the stated conditional null distribution. To meet a budget $b_j$, use a cutoff at least as large as $z_{1-b_j}$. If the cutoff is rounded for implementation, round in the conservative direction and verify the resulting tail probability; a rounded cutoff that is slightly too low can exceed the budget.

Worked Example: Two Continuation Options at One Interim State

Continue the normal example at $Z_1=1.000$, where $C(1)\approx0.0382$. Suppose the prespecified allocation gives option A a budget of 0.0200 and option B a budget of 0.0100. Their combined allocation is 0.0300, below the available conditional error of about 0.0382.

Option A rejects if $Z_2>2.060$. Under the assumed standard normal null distribution, $$ P_{H_0}(Z_2>2.060)=1-\Phi(2.060)\approx0.01970\le0.0200. $$

Option B rejects if $Z_2>2.330$. Its conditional rejection probability is $$ P_{H_0}(Z_2>2.330)=1-\Phi(2.330)\approx0.009903\le0.0100. $$

Both options meet their allocated budgets, so either is valid at this interim state. The numerical cutoffs are conservative to the shown precision: the calculated tail probabilities are below, not above, their budgets. For a complete design, this check must be made across the full range of interim states. For example, the budgets could be defined by fixed shares of $C(x)$ at every $x$, and the option-specific cutoffs recalculated from those state-dependent budgets.

The selection rule may use the interim information. If the trial selects A when an interim operational indicator has one value and B otherwise, the rule is acceptable provided that the indicator is included in $\mathcal F$ and the relevant conditional error inequalities hold given that information. If an indicator is ancillary and the null distribution of the test statistic is unchanged by it, the candidate checks may be the same for either indicator value. If it changes the applicable conditional distribution, that distribution must be used in the checks.

Why the Menu Preserves the Conditional Constraint

The key point is that the selected option is determined using the interim information, while the conditional error check is made given that information. Write $S$ for the selected option, with $S\in\{1,\ldots,J\}$. The rejection event of the adaptive design is $R_S$. Conditional on $\mathcal F$, the selected option is known, so its conditional rejection probability is the conditional rejection probability of that option. The following argument gives the preservation result.

Specify the interim information and selection rule.
Require that $S$ is determined by $\mathcal F$, before continuation outcomes are observed.

Check every candidate conditionally.
For every $j$ and every interim state, verify $C_j(\mathcal F)\le b_j(\mathcal F)\le C(\mathcal F)$.

Apply the check to the selected option.
Because $S$ is known given $\mathcal F$, the selected option’s conditional rejection probability is at most its own budget, hence at most $C(\mathcal F)$.

Average over interim states.
The conditional-expectation identity bounds the unconditional rejection probability by the original design’s rejection probability.

For completeness, the inequality in the final step follows by taking expectations of the conditional bound:

\begin{aligned} P_{H_0}(R_S) &=E_{H_0}\!\left[P_{H_0}(R_S\mid\mathcal F)\right]\\ &\le E_{H_0}\!\left[C(\mathcal F)\right]\\ &=E_{H_0}\!\left[P_{H_0}(R_0\mid\mathcal F)\right]\\ &=P_{H_0}(R_0). \end{aligned}

The first and last equalities are the law of total probability, expressed as conditional expectation. Thus the adaptive design’s null rejection probability is no greater than that of the original design. If the original design controls type I error at $\alpha$, the adaptive design does too. This argument depends on the pointwise conditional bound, not merely on a favorable average across menu options.

Worked Example: Selecting an Option by an Interim Rule

At $Z_1=1.000$, suppose a prespecified interim operational indicator $G$, included in $\mathcal F$, selects option A when $G=1$ and option B when $G=0$. Use the cutoffs from the preceding example. If $G=1$, the conditional rejection probability is approximately 0.01970, below A’s 0.0200 budget. If $G=0$, it is approximately 0.009903, below B’s 0.0100 budget. In both cases, the selected option’s conditional rejection probability is also below the original $C(1)\approx0.0382$.

The choice need not be independent of the interim data. What matters is that it is made using information in $\mathcal F$ and that the inequalities have been verified at each state where each option can be selected. If the selection is randomized using a prespecified random mechanism, condition on the randomization information as well, or average the candidate conditional rejection probabilities using the known selection probabilities. The same bound follows if every possible selected option is within the applicable conditional budget.

Diagnostics, Alternatives, and Limitations

A menu-based design is only as reliable as its conditional calculations. Before implementation, verify that the candidate rejection probability is calculated under the correct null distribution, conditional on all information used for adaptation. Check every option and relevant interim state, including states near the edges of the allowed range. For numerical cutoffs, compute the resulting tail probability after rounding; do not infer validity from a rounded quantile alone. A probability above its allocated budget, even by a small amount, fails that allocation.

One alternative is to give every candidate the full original conditional error, requiring $C_j(\mathcal F)\le C(\mathcal F)$. This is a valid and often simpler sufficient rule, but it does not divide the available conditional error among options. A shared allocation $b_j=w_jC$ is more restrictive and may force more conservative continuation boundaries; in return, it provides explicit budgets whose total is bounded by the original conditional error. Neither approach permits an uncalibrated candidate to be made valid merely by saying that it will be selected infrequently.

The argument also has boundaries. The menu and its conditional error checks must cover the actual continuation procedures. A selection rule that uses future continuation outcomes is not an interim-measurable choice, so the pointwise argument above does not establish error control for it. Likewise, an option calibrated under one conditional null distribution may not be valid if the actual design changes the information used by the test or the distributional assumptions. Those changes require recalculating the relevant conditional probabilities rather than reusing a cutoff without justification.

Common pitfall: “Each option is close to its budget, and the average across the menu is below the original conditional error, so the selection is safe.” This reasoning is not sufficient: the selection rule can favor an option whose conditional rejection probability exceeds the permitted bound at the states where it is chosen. The correction is to verify $C_j(\mathcal F)\le b_j(\mathcal F)\le C(\mathcal F)$ for every selectable option and every relevant interim state. Also verify tail probabilities after rounding; a cutoff that is too low can exceed its assigned budget.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

In the normal example with $c=1.9600$, $Z_1=1.000$, and $Z_2$ conditionally standard normal, write the conditional rejection event and calculate the original conditional error.
At that interim state, option A has budget 0.0200 and uses cutoff 2.060. Does it meet its budget? Give the calculated tail probability.
Explain why checking only the average rejection probability over all options is insufficient when the option is selected using interim information.
State a sufficient condition for a menu of options to preserve the original unconditional type I error.

Solutions. (1) The original rejection event conditional on $Z_1=1.000$ is $Z_2>(1.4142)(1.9600)-1.000=1.7718$. Its conditional error is $1-\Phi(1.7718)\approx0.0382$. (2) $1-\Phi(2.060)\approx0.01970$, which is below 0.0200, so A meets its budget. (3) The selection rule can favor an option with excessive conditional rejection probability in the states where it is selected; an average over options does not rule that out. Check each selectable option at each relevant interim state. (4) If the selection is determined by interim information $\mathcal F$, and every candidate satisfies $P_{H_0}(R_j\mid\mathcal F)\le b_j(\mathcal F)\le C(\mathcal F)$ pointwise, then conditional expectation gives $P_{H_0}(R_S)\le P_{H_0}(R_0)$. Hence an original design controlling type I error at $\alpha$ yields an adaptive design that also controls it at $\alpha$.