Why a Misspecified Adaptation Rule Matters
An adaptive trial’s confidence interval can depend on what happened at an interim analysis: whether the trial continued, which treatment was retained, or how recruitment was changed. Its coverage guarantee is therefore tied to a model of that decision process. If the analysis assumes one interim rule but the trial used another, the reported interval may not have its intended coverage, even when the outcome model is otherwise correct.
The key task is not to declare an interval invalid merely because two rules differ. Instead, identify how the mismatch can affect the probability that the interval contains the true parameter, and quantify that effect under explicit assumptions. A bound can be useful even when the exact coverage under the actual rule is difficult to calculate, but its assumptions must match the trial’s design and analysis.
Coverage Depends on Both the Path and the Data
Let \(D\) denote the adaptation path, such as \(D=1\) for continuation and \(D=0\) for stopping. Write \(C_D\) for the interval reported on path \(D\). Under a specified rule and probability model, coverage can be decomposed as
This identity is a direct partition of the coverage event by the possible paths. It shows two distinct ways misspecification can matter: the rule can change the probability of each path, and it can change the conditional distribution of the data given a path, hence the conditional coverage. Merely comparing continuation probabilities addresses the first component, not automatically the second.
A useful diagnostic is to calculate or estimate the path probabilities and path-specific conditional coverages separately under both rules. If the conditional distributions are unchanged and only the path probabilities differ, the coverage change can be computed by reweighting the same conditional coverages. If conditional distributions also differ, that reweighting alone is not valid.
Worked Example: Reweighting Path-Specific Coverage
Suppose a hypothetical interval analysis is calibrated under an assumed rule that continues with probability \(0.50\). Exact calculations under that rule give conditional coverage \(0.90\) on the continuation path and \(1.00\) on the stopping path. These are illustrative values, not findings from a real trial.
Under the assumed rule, the overall coverage is
Now suppose the actual rule continues with probability \(0.40\). If, and only if, the two rules leave the conditional data distributions and the path-specific intervals unchanged, the corresponding coverage is
That reweighting predicts \(96\%\) coverage in this specified example. In a real adaptive design, the changed rule may also change conditional distributions. For instance, different continuation criteria select different interim observations, so conditional coverage values may change as well. In that case the actual-rule conditional coverages must be evaluated, not borrowed from this calculation.
A Bound Based on Rule Disagreement
A general sensitivity result is available when the two rules can be compared on the same underlying random data. Imagine a common probability space containing the interim information and any potential later outcomes needed to define either rule’s interval. Let \(D_A\) and \(D_B\) be the decisions from the assumed and actual rules on this common data, and let \(I_A\) and \(I_B\) be their coverage indicators: each equals one if its interval contains \(\theta\), and zero otherwise.
Assume that both procedures use the same interval construction on a given path and the same potential data under this coupling. If the rules choose the same path, their coverage indicators agree. Therefore, a difference in coverage indicators can occur only when the rules disagree:
Taking expectations and using \(|E(X)|\leq E(|X|)\) gives the coverage bound:
Thus, if the assumed procedure has coverage at least \(1-\alpha\), and the probability of rule disagreement is at most \(\delta\), the actual-rule coverage is at least \(1-\alpha-\delta\). Since a probability cannot exceed one, the full two-sided statement is
when the assumed coverage is exactly \(1-\alpha\). If the assumed coverage is only known to be at least \(1-\alpha\), the lower bound still holds, but the upper bound needs the actual assumed coverage as its starting point. This result is conservative: disagreement does not necessarily cause a coverage failure, so the actual difference can be much smaller than \(\delta\).
Worked Example: Two Interim Thresholds
Consider a hypothetical standardized interim statistic \(Z\sim N(\mu,1)\), where \(\mu\) is the true standardized interim mean. The assumed rule continues if \(Z>c_A\), with \(c_A=0.5\); the actual rule continues if \(Z>c_B\), with \(c_B=0.7\). Suppose \(\mu=0.6\). The decisions disagree exactly when \(0.5<Z\leq 0.7\), so
Here \(\Phi\) is the standard normal cumulative distribution function. If the interval has exactly \(95\%\) coverage under the assumed rule, the disagreement bound places the actual coverage between
This is a valid but potentially loose range: it says that rule mismatch alone could reduce coverage as far as about \(87.0\%\) under the stated coupling, not that actual coverage equals \(87.0\%\). The calculation also depends on the specified \(\mu\); a bound meant to hold across parameter values must take the worst case over the relevant range of \(\mu\).
Bounding Disagreement Over a Range of Thresholds
Often the actual threshold is not known exactly but is believed to lie in a plausible range. Suppose the assumed rule continues when \(Z>c_A\), while the actual threshold \(c\) is somewhere in \([c_L,c_U]\), with \(c_L\leq c_A\leq c_U\). For a fixed \(\mu\), the largest disagreement probability over that range occurs at one of the two endpoints, so a sensitivity bound is
If there is no restriction on \(\mu\), an especially simple bound applies to any two thresholds \(c_L\) and \(c_U\). Their maximum disagreement over all \(\mu\) is attained when \(\mu\) is their midpoint. To see why, the disagreement probability is \(f(\mu)=\Phi(c_U-\mu)-\Phi(c_L-\mu)\). Its derivative is \(f'(\mu)=-\phi(c_U-\mu)+\phi(c_L-\mu)\), where \(\phi\) is the standard normal density. The derivative is zero when the two arguments are equally far from zero, which occurs at \(\mu=(c_L+c_U)/2\). Because the normal density decreases with distance from zero, this is the maximum. Setting \(w=c_U-c_L\), the maximum is
This worst-case result is about the probability of different decisions, not a direct calculation of the interval’s actual coverage. It can nevertheless turn a range of plausible thresholds into a uniform coverage bound.
Worked Example: A Range of Plausible Thresholds
Suppose a hypothetical trial’s assumed standardized threshold is \(0.6\), while the plausible actual threshold is anywhere from \(0.4\) to \(0.8\). The threshold range has width \(w=0.8-0.4=0.4\). Without restricting \(\mu\), the maximum probability that a rule at one endpoint disagrees with a rule at the other is
The maximum occurs at \(\mu=(0.4+0.8)/2=0.6\), which is also the assumed threshold in this example. If the procedure has \(95\%\) coverage under the assumed rule and the coupling assumptions hold, the worst-case lower bound is
This says the guarantee may be as low as about \(79.2\%\) under that broad threshold uncertainty. Narrowing the plausible range reduces the bound, but only if the narrower range is defensible on design grounds.
Allowing for Additional Probability-Model Error
A rule mismatch is not always the only departure from the calibration model. For example, a working model for the interim statistic might also be imperfect. One standard way to express a bound on such a discrepancy is total variation distance. For probability laws \(P\) and \(Q\), define
where the supremum is over measurable events \(A\). In particular, if the actual and reference laws for the data used by a fixed interval differ by at most \(\eta\) in total variation, the probabilities of the interval-coverage event under those laws differ by at most \(\eta\). Combining this fact with a rule-disagreement bound \(\delta\) gives a conservative bound \(\delta+\eta\), by applying the triangle inequality to the two sources of change.
Worked Example: Combining Two Sensitivity Allowances
Suppose a hypothetical analysis has \(95\%\) coverage under its reference rule and reference model. A justified coupling bounds rule disagreement by \(\delta=0.04\), and a separate assessment bounds the relevant probability-law discrepancy by \(\eta=0.015\). Then the total allowance is \(0.04+0.015=0.055\), giving the lower bound
The corresponding upper bound is \(0.95+0.055=1.005\), which is capped at one. Thus the sensitivity range is \(0.895\) to \(1.000\). This bound is only as credible as the two inputs: the disagreement probability must be evaluated under the relevant parameter values, and the total variation allowance must cover the data law relevant to the interval. It is not enough to label a model discrepancy “small” without quantifying it.
Diagnostics, Alternatives, and Limitations
A practical assessment begins by writing down the assumed and actual decision rules precisely, including thresholds, timing, and any tie-breaking conventions. Then identify the data needed to evaluate their disagreement and determine whether the same potential outcomes and path-specific interval construction can reasonably be used for both rules. If changing the rule changes later allocation or sample size, model the resulting data distributions explicitly or state a separate probability-law bound.
A useful comparison is between three approaches:
| Approach | What it provides | Main limitation |
|---|---|---|
| Evaluate coverage under the actual rule | A direct coverage calculation or simulation for the implemented design. | Requires a sufficiently specified and credible model for the actual rule and outcomes. |
| Use a disagreement or probability-law bound | A conservative coverage range over specified deviations. | Can be wide, and depends on defensible bounds and a valid coupling or model comparison. |
| Use an interval simultaneous across paths | Protection across the specified collection of adaptation paths. | May be wider and only protects the paths and assumptions included in its construction. |
The earlier material on simultaneous inference across adaptation paths provides a relevant alternative when protection across a specified set of paths is the objective. A sensitivity bound answers a different question: how much coverage could change under a stated deviation from the assumed rule? Neither method automatically covers paths or probability models that were not included in its assumptions.
Several warning signs deserve particular attention. A bound based only on the change in continuation probability is inadequate if conditional outcome distributions also change. A threshold calculation based on \(Z\sim N(\mu,1)\) is not justified if the standardized interim statistic is poorly calibrated or has a different variance. Finally, a lower bound below the desired coverage level is not repaired by describing it as conservative: it means the stated sensitivity allowance is too large to certify the target guarantee.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- An interval has \(95\%\) coverage under an assumed rule. Under a valid common-data coupling, the probability the assumed and actual rules disagree is at most \(0.03\). What lower coverage bound follows?
- For \(Z\sim N(\mu,1)\), two continuation thresholds are \(0.2\) and \(0.6\). What is the maximum disagreement probability over all \(\mu\)?
- Why might reweighting the assumed rule’s path-specific conditional coverages fail after changing the interim threshold?
- If the reference model has coverage \(0.95\), the rule-disagreement bound is \(0.02\), and the total variation bound is \(0.01\), what lower coverage bound follows?
Solutions. (1) The lower bound is \(0.95-0.03=0.92\), or \(92\%\), provided the coupling conditions hold. (2) The width is \(0.6-0.2=0.4\), so the maximum is \(2\Phi(0.2)-1\approx 0.1585\). (3) A changed threshold selects a different set of interim data into each path; therefore the conditional distribution, and hence conditional coverage, may change. (4) The combined allowance is \(0.02+0.01=0.03\), giving \(0.95-0.03=0.92\). These are sensitivity bounds, not necessarily the actual coverage.