Bias Correction in Multi-Look Designs

Why Multi-Look Bias Must Be Aggregated Across Stages

When an analysis can stop at several looks, the reported estimate is selected by a stopping rule. Each possible stopping stage selects a different part of the estimator's sampling distribution. An estimate reported at an early efficacy stop, for example, comes from the upper tail of the early-look distribution; an estimate reported at the final look comes from the distribution conditional on not having stopped earlier. The total bias must account for all these possibilities, not just the stage that happened to occur.

The earlier material on unconditional bias correction and analytic bias correction for one-look designs establishes the basic idea of subtracting a design-specific bias. Here the key extension is to calculate that bias across a partition of stopping stages. The probabilities of reaching each stage and the truncated estimate distributions at those stages both matter.

The summand for stage \(k\) is the stage probability multiplied by the conditional bias among trials that stop at that stage. A conditional bias can be large even when its weighted contribution is small, if that stage is rare. Contributions can also have opposite signs and partially cancel. Consequently, neither the largest conditional bias nor the bias at the observed stage alone determines the unconditional bias.

Model and Assumptions for the Calculation

For a concrete derivation, suppose the estimator at look \(k\) follows a canonical normal cumulative-information model. The parameter \(\theta\) is a treatment effect in its stated measurement units, and \(I_k\) is the information available at look \(k\), with \(0<I_1<\cdots<I_K\). Assume

Thus \(\hat{\theta}_k\) is unbiased before applying the stopping rule, is normal with variance \(1/I_k\), and estimates at different looks are correlated because they use accumulated data. This model is appropriate when the information levels and normal approximation are adequate, the parameter is stable over the trial, and the stopping boundaries and rules are specified. The calculation must use the actual rule: for example, whether futility stopping is binding changes the stopping events and therefore can change the bias.

Let the stopping rule define \(A_k\) using the statistics observed up to look \(k\). For an upper efficacy boundary at each nonterminal look and a mandatory final analysis, \(A_k\) means crossing the boundary at look \(k\) without crossing an earlier boundary, for \(k<K\); \(A_K\) means reaching the final look without an earlier stop. To calculate the bias, one needs both \(P_\theta(A_k)\) and the conditional mean of \(\hat{\theta}_k\) over that event. These are probabilities and means under the joint normal model, restricted to the relevant stopping region.

Deriving the Stage-Weighted Bias

The sum in the definition follows by partitioning the expectation according to the stopping stage. On event \(A_k\), the reported estimate is \(\hat{\theta}_k\). Therefore

The first equality uses the fact that exactly one of the mutually exclusive events \(A_k\) occurs, and the second is the definition of conditional expectation. Although each fixed-look estimator has mean \(\theta\) without selection, its conditional mean given \(A_k\) generally differs from \(\theta\). That difference is precisely why the stage contributions need to be calculated from truncated, rather than unselected, distributions.

Two-Look Normal Design: A Complete Calculation

Consider a two-look design with information \(I_1=25\) and \(I_2=100\). At the first look, stop for efficacy if \(Z_1\geq c\), where \(Z_1=(\hat{\theta}_1-\theta)\sqrt{I_1}\) is considered here under the boundary calculation at \(\theta=0\), and let \(c=2\). Otherwise, continue to the final look. The final estimator is \(\hat{\theta}_2\). Assume the stated normal cumulative-information model, known information levels, and no other early stopping rule.

At \(\theta=0\), \(Z_1\sim N(0,1)\), so the first-stage stopping probability is \(p_1=1-\Phi(2)=0.02275\), and the continuation probability is \(p_2=\Phi(2)=0.97725\). Write \(\phi\) and \(\Phi\) for the standard normal density and distribution functions. The correlation between the standardized errors at the two looks is \(\rho=\sqrt{I_1/I_2}=0.5\).

For a standard normal variable truncated above \(c\), its conditional mean is \(\phi(c)/[1-\Phi(c)]\); truncated below \(c\), it is \(-\phi(c)/\Phi(c)\). These follow by integrating \(z\phi(z)\), using \(\phi'(z)=-z\phi(z)\). Since \(\hat{\theta}_1-\theta=Z_1/\sqrt{I_1}\), the first-stage conditional bias is \(0.2\phi(2)/p_1\). For the second stage, the standardized final error has conditional mean \(\rho E[Z_1\mid Z_1<2]\), by the conditional expectation formula for jointly normal variables. The stage contributions are therefore:

Adding the contributions gives \(b(0)=0.01080-0.002700=0.008100\) units. Equivalently, the algebra combines to

The two calculations agree up to rounding. The first-stage conditional bias is much larger than the total bias, because the early stop is uncommon and the continuation-stage contribution is negative. This result illustrates why stage-specific conditional bias cannot be substituted for the design-wide unconditional bias.

Turning the Bias Calculation Into a Correction

The bias function is generally a function of the unknown parameter: the chance of crossing a fixed boundary and the conditional distributions both change with \(\theta\). One design-based correction is to calibrate the observed estimator against its expectation under the stopping rule. Define \(m(\theta)=\theta+b(\theta)\). If \(m\) is one-to-one over the parameter range of interest, an implicit bias-corrected estimate solves \(\hat{\theta}_T=m(\tilde{\theta})\). A simpler one-step plug-in correction is \(\hat{\theta}_T-b(\hat{\theta}_T)\).

Neither calculation makes the corrected estimator exactly unbiased in every finite sample: the bias function is evaluated at an unknown value or an estimate of it, and nonlinear correction can introduce further bias. The correction instead uses the specified stopping mechanism to reduce the known selection component. The implicit solution can be preferable when the mean map is monotone and numerical root-finding is stable; the plug-in version is easier but can be inaccurate when the bias function changes rapidly.

Diagnostics, Alternatives, and Limitations

A reliable multi-look correction depends on correctly describing the entire stopping process. Useful checks include confirming that the stage probabilities sum to one, checking that each stage event includes all required non-stopping conditions from earlier looks, and verifying that conditional means were computed from the distribution restricted to that event. For normal-model calculations, compare numerical integration with simulation from the joint distribution under the same information schedule and stopping boundaries. Large disagreement can signal an incorrectly implemented boundary, an incorrect covariance structure, or a numerical integration problem.

The most relevant alternative is a correction conditional on the observed stopping stage. That answers a different bias question: it uses \(E_\theta[\hat{\theta}_k-\theta\mid A_k]\) for the realized stage \(k\), rather than averaging the contributions over all possible stages. A stage-conditional correction may be appropriate when the target is explicitly conditional on that stage; it is not a substitute for unconditional correction. Another option is to use simulation under a fully specified design to estimate stage probabilities and truncated means. Simulation can handle complicated stopping regions, but it introduces Monte Carlo error and still requires the model and stopping rule to be correctly specified.

Common failure modes include using independent-look probabilities despite the cumulative-data correlation, omitting a stage or counting a trial in more than one stopping event, and using the bias for one boundary configuration with a different design. A correction can also be unstable if \(m(\theta)\) is nearly flat or nonmonotone, because inversion may be imprecise or have multiple solutions. Finally, the normal known-information model may not represent a design with an estimated standard error, changing treatment effects, or a different estimator. In those settings the stage probabilities and conditional distributions must be recalculated under a suitable model; the formulas above do not automatically carry over.