Worked Inference for a Two-Stage Adaptive Trial

Why Interim Selection Changes Final Inference

In a two-stage adaptive trial, interim data may determine which treatment continues. That decision changes the distribution of the data used in the final analysis: the selected treatment's first-stage estimate is not an ordinary, unselected estimate. Pooling it with second-stage data and applying a standard interval formula can therefore give misleading inference, even when the arithmetic and the assumed normal model are otherwise correct.

We will work with a deliberately simple design so the issue is visible. There are two candidate treatments, A and B, and a control. At the interim analysis, the trial selects the candidate with the larger estimated mean benefit. A second-stage comparison is then conducted for the selected candidate. The goal is to estimate the mean benefit of the treatment actually selected, in the trial's outcome units.

The independence and fixed second-stage plan are important here. They make the second-stage estimate independent of the data that chose the treatment. The known-variance assumption permits normal critical values without estimating a variance from the same data. If the design instead changes sample size, uses shared controls, stops early, or uses estimated variances, the calculations below do not automatically apply; the analysis must represent those features.

Worked Trial: Selection, Pooling, and a Valid Conditional Interval

Suppose the outcome is reduction in systolic blood pressure, measured in mm Hg, so a positive treatment effect means a larger mean reduction than control. Each stage compares each candidate with its own independent control group. Assume the within-group outcome standard deviation is known to be \(4\) mm Hg, with \(32\) participants per group in each comparison and stage. Thus, for either candidate and either stage,

Why Selection Can Bias the Pooled Estimate

The direction of the selection effect is easiest to see when the candidates have equal true effects and equal first-stage variances. Write their first-stage estimates as \(X_{A1}=\theta+\varepsilon_A\) and \(X_{B1}=\theta+\varepsilon_B\), where the errors are independent \(N(0,\tau^2)\). The selected first-stage estimate is their maximum. Using \(\max(U,V)=(U+V+|U-V|)/2\), and \(X_{A1}-X_{B1}\sim N(0,2\tau^2)\), gives

The selected first-stage estimate is thus upward on average in this equal-effect setting. Since the second-stage estimate is independent and unbiased for the selected treatment's effect, an equally weighted pooled estimate has expected selection bias \(\tau/(2\sqrt{\pi})\). With \(v_1=v_2=1\), \(\tau=1\), so this illustrative expected bias is \(1/(2\sqrt{\pi})=0.2821\) mm Hg.

This derivation is not a universal bias correction. If effects differ, variances differ, or the selection rule is more complex, the amount and even the direction of bias need not match this simple result. Its purpose is to show why the fixed-selection normal calculation cannot simply be carried over after choosing the apparent winner.

A Stage-Two-Only Interval With Conditional Coverage

A direct solution is to base the final interval on the independent second-stage estimate alone. It may be less precise than a correctly calibrated method that uses both stages, but its coverage justification is particularly transparent under the stated design.

has conditional coverage \(1-\alpha\) for \(\theta_J\). Here \(z_{1-\alpha/2}\) is the \(1-\alpha/2\) quantile of the standard normal distribution.

To justify the result, condition on the first-stage data. This fixes the identity \(J\), while independence leaves the second-stage estimate normally distributed with mean \(\theta_J\) and variance \(v_2\). Standardization therefore gives

Rearranging the inequalities yields the stated interval. This is a conditional statement about repeated second-stage outcomes, given the observed first-stage data and resulting selection. It does not say there is a 95% probability that a fixed parameter lies in the realized interval.

Other Selection-Aware Approaches

Discarding the first stage is not the only option. Which method is appropriate depends on whether the main goal is conditional coverage for the selected treatment, familywise protection across candidates, or greater precision under a fully specified selection model.

For example, a Bonferroni simultaneous procedure for two first-stage effects constructs each interval at confidence level \(1-\alpha/2\). By the union bound, the probability that either interval misses its own effect is at most \(\alpha\). Therefore, the interval corresponding to whichever candidate was selected is covered whenever both intervals cover. With \(SE=1\) mm Hg and \(\alpha=0.05\), the normal critical value is approximately \(z_{0.975}=1.96\) for each 97.5% two-sided interval only if each tail is allocated carefully: a 97.5% two-sided interval uses \(z_{0.9875}=2.241\). Candidate A's simultaneous interval here is \(1.5\pm2.241=[-0.741,3.741]\) mm Hg. This is conservative protection based on the first stage, not a conditional 95% interval given selection.

A selective likelihood takes the selection event into account directly. For a full first-stage data vector \(x\) and a selection region \(R_j\) in which candidate \(j\) wins, the conditional density has the form

The denominator depends on the unknown effects and the selection rule. It cannot generally be omitted or treated as a constant. Constructing intervals from this likelihood requires specifying the joint model, handling nuisance effects, and inverting a calibrated test or likelihood procedure. Merely truncating the selected estimate at the observed competitor estimate, without deriving the appropriate conditional distribution, is not a valid general solution.

Diagnostics, Failure Modes, and Limits

Before reporting an interval, verify that the analysis matches the actual adaptation. Record the selection rule, the candidate estimates used at interim, the planned second-stage allocation, and any deviations. Check whether the outcome model is reasonably normal at the estimator level, whether the variance used is known or properly estimated, and whether second-stage data are genuinely independent of the selection data. Shared controls, outcome-dependent continuation, an unrecorded selection threshold, or changing the sample size after seeing interim results can invalidate the simple conditional argument.

• Ignoring the selection step: Calling the pooled interval an ordinary 95% interval assumes a fixed candidate. Report it as naive unless its coverage has been established under the complete adaptive design.
• Confusing conditional and unconditional coverage: The stage-two-only interval has conditional coverage given the first-stage data under this model. A simultaneous procedure instead gives joint protection over candidates; these are different guarantees.
• Forgetting the target: \(\theta_J\) is the effect for the treatment selected by the rule, not the effect of a candidate chosen in advance. State that target explicitly.
• Using an unplanned second-stage sample size: The displayed standard error and conditional argument assume the planned \(v_2\). If the sample size changes based on interim information, the method needs to account for that rule.

No single interval is best for every objective. Stage-two-only inference is simple and robust to selection under the stated independence conditions, but may be inefficient. Simultaneous intervals protect a family of effects but may be conservative. Selective methods can use more information but require a more demanding model and correct implementation. The naive interval is easiest to calculate, yet its familiar standard error does not repair the selection it ignores.