Why the Interim Detail Matters
An adaptation choice is often described in broad terms: continue as planned, increase the sample size, or switch to a different continuation design. But the choice usually follows from interim data. Two patients—or two possible interim data histories—can lead to the same named adaptation while leaving different amounts of type I error available under the original design. Averaging those histories together can make a proposed continuation look acceptable even when it spends too much error for some histories.
The distinction is between a conditional error function evaluated at the interim information and an average conditional error over a group of interim outcomes. The first is the relevant object for a general preservation argument. A category average can be informative for a fixed design under a specified null distribution, but it does not show that the error bound is respected at each interim result.
In a continuous-data design, conditioning on \(X=x\) is understood through a conditional distribution, or equivalently through a version of the conditional expectation defined almost surely. The main idea is unchanged: retain the interim information that determines the original design's remaining rejection probability. The calculation below uses discrete interim states to make every probability visible.
The Pointwise Preservation Principle
Suppose the original design has type I error at most \(\alpha\) under \(H_0\). A proposed continuation design, after observing \(X=x\), has conditional rejection probability \(e_1(x)\). A direct sufficient condition for preservation is \[ e_1(x)\le e_0(x) \] for every interim result that can occur under \(H_0\), apart from probability-zero sets. This compares the new design's rejection probability with the original design's remaining error at the same interim history.
The first and third equalities are the law of total probability, written as conditional expectation. The inequality follows by integrating the pointwise bound. Thus the new design controls the overall type I error whenever the original design did. This justification does not require the new continuation to have the same rejection probability as the original one; using less conditional error is allowed.
A category-level comparison, \(\bar e_1(c)\le\bar e_0(c)\), is weaker. It controls only an average under the distribution of interim results used to calculate that average. If the conditional rejection probability is larger than \(e_0(x)\) for some interim results, the pointwise argument above no longer applies. Whether the fixed design still has acceptable marginal type I error then depends on the relevant distribution and weighting of those results; the category average alone is not a general preservation guarantee.
Worked Example: Equal Category Averages, Different Conditional Errors
Consider a hypothetical two-stage trial. Under \(H_0\), the interim result is either \(L\) or \(H\), each with probability \(0.50\). Both results lead to adaptation category \(C=A\). For simplicity, suppose the stage-two statistic gives a \(p\)-value that is Uniform(0,1) under \(H_0\), conditionally on either interim state. The original design rejects if that \(p\)-value is below a state-specific threshold.
Let the original thresholds be \(0.04\) after \(L\) and \(0.06\) after \(H\). Uniformity gives \(e_0(L)=0.04\) and \(e_0(H)=0.06\). Its category-average error is \[ \bar e_0(A)=0.50(0.04)+0.50(0.06)=0.020+0.030=0.050. \]
Now consider a proposed continuation with thresholds \(0.01\) after \(L\) and \(0.09\) after \(H\). Its category average under the same equal weighting is \[ \bar e_1(A)=0.50(0.01)+0.50(0.09)=0.005+0.045=0.050. \] The averages match exactly. But after observing \(H\), the new conditional rejection probability is \(0.09\), which exceeds the original conditional error \(0.06\) by \(0.03\), or 3 percentage points. The category average has hidden that local excess.
For this exact hypothetical design, with the same null probabilities for \(L\) and \(H\), the proposed design's marginal type I error is still \(0.050\). That fact follows from the particular fixed weights and the calculation above; it does not establish pointwise conditional-error preservation. In particular, it cannot justify the proposed continuation when the relevant within-category weights differ or when the adaptation procedure uses interim information not represented by the category label.
How Averages Can Fail as a Preservation Argument
A category average implicitly uses weights. In general, write \(\pi_x=P_{H_0}(X=x\mid C=c)\). Then \[ \bar e(c)=\sum_{x:g(x)=c}\pi_x e(x). \] If a calculation reports only \(\bar e(c)\), it does not reveal the individual \(e(x)\), and it relies on the weights \(\pi_x\). If the interim composition within a category changes, or if the average was calculated with the wrong null distribution, the reported average need not describe the actual conditional rejection probability of the adapted procedure.
The next example makes the weighting issue concrete. It does not say that every change in the interim mix violates type I error control. Rather, it shows why an average computed for one mix cannot be used as a general bound for another mix.
Worked Example: A Changed Within-Category Mix
Use the same two interim states and the same Uniform(0,1) stage-two \(p\)-value. Suppose the original continuation thresholds are \(0.02\) after \(L\) and \(0.06\) after \(H\). Under the reference null distribution, \(P(L\mid A)=P(H\mid A)=0.50\), so the original category average is \[ \bar e_0(A)=0.50(0.02)+0.50(0.06)=0.010+0.030=0.040. \]
A proposed continuation uses thresholds \(0.005\) after \(L\) and \(0.075\) after \(H\). Its average under those same reference weights is \[ \bar e_1(A)=0.50(0.005)+0.50(0.075)=0.0025+0.0375=0.040. \] So a comparison based only on the reference category averages would report no increase. Pointwise, however, the proposed error after \(H\) is \(0.075\), exceeding the original value \(0.06\).
Now suppose the relevant null mix within category \(A\) is instead \(P(L\mid A)=0.25\) and \(P(H\mid A)=0.75\). Under that mix, the original continuation has error \[ 0.25(0.02)+0.75(0.06)=0.005+0.045=0.050, \] whereas the proposed continuation has error \[ 0.25(0.005)+0.75(0.075)=0.00125+0.05625=0.0575. \] The new procedure is 0.0075 higher than the original continuation under this mix. The \(0.040\) average calculated with equal weights did not protect against that change.
The mix in an actual trial is determined by its null model and adaptation rule; it should not be changed casually in an analysis. This example instead identifies the logical gap: an average under one stated distribution is not a pointwise bound and is not automatically valid under another distribution or rule. If the design is fixed and the correct within-category null distribution is known, an average can support a marginal calculation for that design. It cannot replace checking the conditional errors when the goal is a general conditional-error preservation claim.
What to Check in a Two-Stage Design
For each possible interim history, the original conditional error describes the probability of eventual rejection under \(H_0\), given that history and the original continuation. The proposed continuation must be evaluated on the same conditioning information. A category label is sufficient only if it retains all information needed to compare those conditional rejection probabilities. If two distinct histories in one category have different original conditional errors, collapsing them can lose relevant information.
State exactly what is observed before adaptation, including the statistic, its possible values or range, and any other data used to make the choice.
For each relevant interim result \(x\), compute \(P_{H_0}(R_0\mid X=x)\) using the original continuation rule and its null distribution.
Compute \(P_{H_0}(R_1\mid X=x)\), using the conditional distribution appropriate after the observed interim data.
Check whether \(e_1(x)\le e_0(x)\) across the relevant histories. A category average can summarize the result, but should not conceal a failed pointwise comparison.
If exact pointwise control is too conservative or difficult to achieve, a worst-case bound is another cautious option: require the largest proposed conditional error over all histories in a category to be no larger than the smallest original conditional error there. This is stronger than necessary, but it does not depend on the within-category weights. An average-based argument is less conservative for a fixed, well-specified design, but its validity depends on the exact distribution used and does not provide the same history-by-history guarantee.
Worked Example: A Pointwise Check of a Proposed Continuation
Suppose the original conditional errors are \(e_0(L)=0.04\) and \(e_0(H)=0.06\), as in the first example. A proposed continuation has conditional rejection probabilities \(e_1(L)=0.03\) and \(e_1(H)=0.06\). The checks are \[ e_1(L)=0.03\le 0.04=e_0(L),\qquad e_1(H)=0.06\le 0.06=e_0(H). \]
Both states satisfy the pointwise condition. With equal null probabilities, the proposed marginal type I error is \[ 0.50(0.03)+0.50(0.06)=0.015+0.030=0.045, \] which is no greater than the original value \(0.050\). More generally, the pointwise inequalities ensure the integrated error bound for any null probabilities assigned to these states, because each term in the weighted average is bounded separately.
Diagnostics, Alternatives, and Limitations
A practical warning sign is a conditional-error table or plot that reports only one value per adaptation category even though interim results within that category lead to different original continuation probabilities. Inspect the unaggregated values, and verify the null distribution and weights used in any average. For a continuous interim statistic, examine the function over its relevant range, including boundary regions where conditional error can change rapidly. Numerical integration can check an average, but it cannot by itself establish a pointwise inequality.
Three approaches answer different questions:
| Approach | What it checks | Main limitation |
|---|---|---|
| Pointwise conditional-error comparison | Whether the proposed rejection probability is bounded at each interim history | May be conservative or computationally demanding for complex histories |
| Category-average comparison | Whether average error is bounded under a specified conditional null distribution | Can conceal local excess and depends on the stated weights |
| Worst-case comparison | Whether the largest proposed error is bounded by a conservative original bound across a category | Can be substantially more conservative than a history-specific comparison |
The pointwise condition is a sufficient preservation argument, not a claim that every pointwise excess must inflate the marginal type I error of every fixed design. As the first example shows, equal and unchanged weights can leave the overall error unchanged despite a local excess. The limitation is one of what has been established: matching one average proves at most a statement about that average under its specified assumptions. It does not establish the stronger conditional-error bound needed to justify preservation across interim histories.
A common mistake is to reason, “The average conditional error in this adaptation category is unchanged, so the conditional-error principle is satisfied.” The correction is to ask what is being conditioned on. If the preservation claim is conditional on the observed interim data, compare the new and original errors at that level. If only a category average is available, state the exact null distribution and category rule that make the average relevant, and do not present it as a pointwise guarantee.
Check Your Understanding
Answer each question, then compare your reasoning with the short solutions.
- Under \(H_0\), two interim states have probabilities \(0.40\) and \(0.60\). A proposed continuation has conditional errors \(0.02\) and \(0.08\). Compute its category-average error.
- The original conditional errors for those states are \(0.03\) and \(0.07\). Does the proposed continuation satisfy pointwise conditional-error preservation?
- Can equality of the original and proposed category averages, under unchanged known null weights, establish equal marginal type I error for that fixed two-state design? What does it fail to establish?
- Why is a numerical integration of the category-average error not enough to verify a pointwise conditional-error bound?
Solutions. (1) The average is \(0.40(0.02)+0.60(0.08)=0.008+0.048=0.056\), or 5.6%. (2) No. The first state satisfies \(0.02\le0.03\), but the second fails because \(0.08>0.07\). (3) Yes, if the weights and conditional probabilities are correct and unchanged, equality of the weighted averages gives equal marginal rejection probabilities for that fixed design. It does not establish a pointwise bound or a guarantee under different weights or histories. (4) Integration returns a weighted average; positive and negative differences across histories can cancel, so an average does not show the inequality holds at every interim result.