Naive Wald Intervals After Adaptation

Why Adaptation Can Undermine a Familiar Interval

A conventional Wald confidence interval is easy to compute and easy to report. Its simplicity can be misleading in an adaptive trial: if the observed data determine which treatment is reported, whether the trial stops, or how much additional information is collected, the reported estimate is not necessarily distributed as it would be under a fixed design. The interval’s center may be systematically displaced by the adaptation rule, and its standard error may not describe the uncertainty in the selected estimate.

This tutorial isolates that issue. The examples use normal models with known standard errors so that the effect of adaptation can be calculated exactly. Real trials often estimate their standard errors and may have non-normal outcomes; those features add uncertainty but do not, by themselves, correct selection or stopping effects.

For example, a 95% Wald interval uses \(z_{0.975}\approx1.96\). If the estimator is approximately normal with mean \(\theta\) and standard deviation accurately estimated by \(\widehat{SE}\), the interval has approximately 95% coverage across repetitions of the specified design. Adaptation matters when the distribution relevant to inference differs from that fixed-design distribution. In particular, selecting an estimate because it is large tends to make the selected estimate too large, while stopping after an unusually favorable interim result tends to make that result unusually favorable.

The phrase naive Wald interval here means an ordinary Wald interval computed from the estimate and standard error as if the adaptation rule had not affected their sampling distribution. It is not a claim that every interval used in an adaptive trial must fail. Whether a conventional interval is valid depends on the adaptation rule and on the joint distribution of the data used for adaptation and estimation.

What the Fixed-Design Calculation Leaves Out

In a fixed design, the usual normal approximation concerns the estimator generated by that design. Under adaptation, the actual reported estimator may instead be a function of both the data and the rule: for example, the estimate from the arm with the largest observed effect, or the interim estimate in trials that cross an efficacy boundary. Even if each candidate estimate separately has a well-behaved fixed-design distribution, the selected or stopping-time estimate need not have that same distribution.

The key point can be seen in a simple normal model. Suppose a standardized estimate \(Z\) has distribution \(N(0,1)\) when the true effect is zero. Without selection, the usual interval covers zero whenever \(-1.96\le Z\le1.96\), an event with probability about 0.95. If the analysis reports the estimate only when \(Z\) is large, it is no longer appropriate to use that unselected probability without considering the reporting rule.

Deriving the Selection Effect

The size of the problem can be quantified in a simple setting. Assume \(K\) independent standardized estimates \(Z_1,\ldots,Z_K\), each distributed as \(N(0,1)\) when all \(K\) true effects are zero. Suppose the reported estimate is the largest, \(M=\max(Z_1,\ldots,Z_K)\), and the naive interval for the selected effect is \(M\pm1.96\) in standardized units. Because all true effects are zero, the interval covers the true effect exactly when \(-1.96\le M\le1.96\).

Write \(\Phi\) for the standard normal cumulative distribution function. Independence gives \(P(M\le1.96)=\Phi(1.96)^K\), since every estimate must be at most 1.96. Similarly, \(P(M<-1.96)=\Phi(-1.96)^K\), since every estimate must be below -1.96. Subtracting the latter event from the former yields the coverage probability under the full selection-and-reporting rule:

This is a derivation for the stated model, not a universal formula for every adaptive trial. It shows why the usual interval can undercover: the chance that at least one candidate estimate lies above the upper confidence limit grows with the number of candidates, while selection of the maximum makes such an estimate more likely to be reported.

Early Stopping Can Produce the Same Problem

Selection need not be among several treatments. In an efficacy-stopping design, the trial may stop when an interim statistic crosses a prespecified boundary. If an ordinary interval is then formed from the same unusually favorable interim estimate, the estimate is being reported conditional on having passed that boundary. The fixed-design Wald calculation does not, on its own, account for that event.

The following calculation conditions on stopping at one specified efficacy boundary. It uses a single interim look and an exact normal model, so there is no approximation in the coverage calculation beyond the choice of a normal model for the standardized estimate.

Assumptions, Diagnostics, and Alternatives

The normal examples rely on explicit assumptions: the standardized estimates have the stated normal distributions; their standard errors are known; independence is assumed in the three-treatment selection example; the true effects are zero for the coverage calculations; and the selection or stopping rule is exactly the one described. These assumptions make the examples transparent. In an actual trial, investigators must use the joint distribution induced by the design, including any correlation among treatment estimates, the interim decision rule, and the final estimator.

Useful diagnostics begin with the protocol and analysis plan, rather than with the interval endpoints. Determine which outcomes or estimates influenced selection, continuation, stopping, sample size, or the choice of analysis population. Then ask whether the reported estimator’s standard error and reference distribution account for those decisions. Simulation under the full design can reveal discrepancies: generate trial data under specified parameter values, apply the adaptation rule in each replicate, calculate the reported interval, and record whether it contains the parameter. A coverage estimate is the number of containing intervals divided by the number of simulated trials, with its own Monte Carlo uncertainty.

Warning signs include an estimate reported only because it crossed a favorable boundary, an interval centered on the winner among multiple candidates, and a standard error calculated as though the selected sample size or analysis had been fixed in advance. Conversely, adaptation does not necessarily cause a problem if the data driving adaptation are independent of the estimator in the relevant model and the analysis preserves that independence. The independence and distributional claims must be justified; they should not be presumed from the word “adaptive.”

The main alternative is to use an inferential procedure calibrated to the design and the reported estimator, rather than to apply the fixed-design Wald formula unchanged. Possibilities include an adaptation-aware likelihood or an interval obtained by inverting tests whose error rates reflect the design. The appropriate construction depends on the adaptation rule and the estimand. The important distinction is conceptual: retaining a familiar center and standard error does not make an interval valid if its sampling distribution has changed.

Common Mistake and Practical Limits

• Mistake: “The standard error is correct for each treatment, so the interval for the selected treatment is automatically a 95% interval.”
• Correction: A standard error for each prespecified estimate does not by itself describe the distribution of the estimate selected for being largest. Selection can alter the center and the tail probabilities relevant to coverage.
• Mistake: “The trial stopped at a prespecified boundary, so the ordinary interval is unaffected.”
• Correction: A prespecified rule can still select outcomes based on their observed values. Prespecification makes the rule reproducible; it does not guarantee that an interval ignoring the rule has its nominal coverage.

These examples establish that naive Wald intervals can undercover, not that they always do. Their exact numerical results depend on the model, the adaptation rule, the number and dependence of candidate estimates, and how the final standard error is calculated. Nor does a coverage failure alone prescribe one universal replacement. It signals that inference must be evaluated under the actual design, with the target parameter and the role of the adaptation rule made explicit.