Type I Error Under Adaptive Allocation

What Does Type I Error Mean in an Adaptive Trial?

A trial’s allocation rule and its analysis rule jointly determine how often the trial reports evidence against a null hypothesis when that null is true. With response-adaptive randomization, later assignments depend on earlier observed outcomes. Repeating the trial therefore repeats not just the outcome sampling, but also a sequence of assignments that reacts to those outcomes.

The relevant error rate is a property of the whole design-and-analysis procedure. It is not a property of the test statistic alone. To assess it, specify the null hypothesis, the outcome-generating model, the allocation algorithm, the stopping and missing-data rules, and the rule for rejecting the null. The earlier material on operating characteristics distinguishes performance averaged over repeated trials from performance conditional on a particular history; the distinction matters here because adaptive histories vary from trial to trial.

This tutorial focuses on a sharp null: every participant would have the same outcome under every treatment. For two treatments, if \(Y_i(A)\) and \(Y_i(B)\) denote participant \(i\)’s potential outcomes, the sharp null is \(Y_i(A)=Y_i(B)\) for every \(i\). It is stronger than a null that says only that two population means are equal: individual treatment effects may differ while the average effect is zero. Under the sharp null, assignment cannot change any participant’s outcome, although outcomes can still vary between participants.

Why Adaptation Can Change Calibration

In a fixed-allocation randomized trial, treatment labels are generated without using participants’ outcomes. Some standard analyses rely on that randomization structure or on a model that correctly describes it. In an adaptive trial, the probability of a later assignment can depend on earlier outcomes. Thus, conditional on the observed outcomes and final arm sizes, treatment labels need not be distributed as if they had been uniformly permuted among participants.

This does not mean that every conventional test is invalid whenever allocation adapts. It means that its calibration must be justified for the actual design and analysis, rather than assumed from a fixed-allocation setting. Depending on the allocation rule, statistic, outcome model, and sample size, a usual test may be conservative, anti-conservative, or adequately calibrated. There is no universal direction of error.

Design-Based Calibration Under the Sharp Null

A direct way to assess a test under a sharp null is to hold the potential outcomes fixed and generate assignments according to the trial’s actual adaptive mechanism. For each possible assignment sequence, the analysis produces a test statistic. The sequence probability is the product of its step-by-step assignment probabilities, evaluated using the history that would have occurred along that sequence.

Here \(\mathbf A=(A_1,\ldots,A_n)\) is an assignment sequence, \(\mathbf Y\) is the vector of outcomes under the sharp null, \(H_{i-1}\) is the assignment-and-outcome history before participant \(i\), and \(q_i(a_i\mid H_{i-1})\) is the probability the design assigns participant \(i\) to \(a_i\) at that history. The formula follows from the chain rule for conditional probabilities. It also makes a practical requirement explicit: when an alternative assignment is considered, later assignment probabilities must be recalculated from the history that alternative would create.

For a test statistic \(T\), a design-based tail probability is calculated by adding the probabilities of assignment sequences for which \(T\) is at least as extreme as its observed value, using the test’s stated extremeness rule. An exact test based on the correct randomization distribution has rejection probability no greater than its nominal level under the sharp null; discreteness can make it conservative. This statement assumes the sharp null is true, the allocation mechanism is correctly specified, and the statistic and rejection rule are fixed as stated.

The last step clarifies two useful checks. A design-based calculation conditional on fixed outcomes examines the randomization error for those outcomes. The unconditional type I error also averages over the way outcomes vary between repeated trials under the null model. If the conditional rejection probability is at most \(\alpha\) for every allowed fixed outcome configuration, the unconditional probability is also at most \(\alpha\): an average of quantities no greater than \(\alpha\) cannot exceed \(\alpha\).

Assumptions, Diagnostics, and Failure Modes

A simulation or design-based calculation is informative only if it represents the trial whose error rate is being evaluated. In particular, check that the outcome model is a legitimate null model for the question, and that the assignment algorithm uses the same information, probabilities, and timing as the intended trial. If the analysis uses a particular estimator, variance estimate, or rejection threshold, the assessment must use that same analysis.

• Verify the sharp null used for assessment. If outcomes are generated under different treatment effects, the calculation is not a type I error assessment for the sharp null.
• Reproduce the entire assignment history. Updating probabilities from a different or incomplete history changes the design being assessed.
• Check positivity and empty arms. If the rule assigns an arm with zero probability at some history, some assignment sequences cannot occur. If an analysis is undefined when an arm has no participants, its handling of that event must be specified.
• Include operational rules. Stopping, exclusions, missing outcomes, and any additional interim decisions can change the probability of rejection if they affect assignments or the analysis.
• Quantify simulation error. Report the number of simulated trials and the estimated Monte Carlo uncertainty. A rate close to \(\alpha\) from a small simulation may be inconclusive.

A short justification for the unconditional check is useful. Let \(r(\mathbf Y)\) be the probability of rejection over assignments for a fixed outcome vector \(\mathbf Y\) under the sharp null. If the design-based test ensures \(r(\mathbf Y)\leq\alpha\) for every allowed \(\mathbf Y\), then \(P(\text{reject})=E\{r(\mathbf Y)\}\leq E(\alpha)=\alpha\). The inequality follows by averaging the conditional rejection probabilities over the null distribution of outcomes. If the bound holds only on average under a particular outcome model, the conclusion is correspondingly limited to that model.

How the Main Alternatives Differ

Design-based assessment is especially useful when the trial’s randomization mechanism is known and the scientific null is sharp. Repeated simulation is useful when the full design is too complicated to enumerate, or when the operating characteristics of a planned procedure must be examined under a specified outcome model. Neither approach rescues an incorrect description of the design. A conventional test may still be appropriate if its calibration is established for the adaptive setting, but it should not be selected solely because its formula is familiar.

Common Mistake

Wrong reasoning: “Under the null, treatments have the same effect, so I can shuffle the final treatment labels uniformly within the observed arm sizes.” This treats equality of outcomes under the sharp null as if it made the adaptive assignment sequence exchangeable after observing outcomes.

Correction: The sharp null makes outcomes invariant to assignment; it does not erase the rule by which assignments depended on earlier outcomes. A design-based reference distribution must use the probabilities generated by the adaptive algorithm, or another analysis must be separately justified for that algorithm. Also distinguish a sharp null from a null about equal average outcomes: the latter alone does not specify every participant’s outcome under every alternative assignment.