Using Later-Stage Outcomes After Interim Selection

Why Later-Stage Outcomes Change the Estimation Problem

When an interim analysis determines which treatment arms continue, the interim estimate for a retained arm is not an ordinary unbiased estimate viewed conditionally on that decision. An arm may continue partly because its interim estimate was unusually high. Outcomes collected after the decision can still provide information about its treatment effect: because those patients enter after selection, their outcomes were not used to choose the arm. The central task is to use that later information without pretending that the selected interim estimate has its original, unconditional distribution.

The setting here is a retained arm with one treatment-effect parameter, denoted \(\theta_j\). At each stage, the effect estimate is a treatment-versus-control contrast. Its variance must account for the trial’s allocation and any within-stage covariance, including covariance from a shared control. For clarity, the formulas below first use a single contrast with known variances. The same logic applies to a vector of arm estimates when the corresponding covariance matrices are used.

These assumptions are substantive. If the treatment effect changes over time, the two stages need not estimate the same parameter. If later outcomes influence which patients are included or how their results are measured, independence from selection may fail. And when a control group is shared across treatment comparisons within a stage, contrasts may be correlated: their covariance cannot be discarded merely because the stagewise estimates have been calculated separately.

What Selection Does to a Pooled Estimate

Write \(b_j(\boldsymbol{\theta})=E(X_j\mid S_j)-\theta_j\) for the interim estimate’s conditional selection bias under a specified selection rule and effect vector \(\boldsymbol{\theta}\). The bias can be positive, negative, or small, depending on the rule and the true effects. Since \(Y_j\) is independent of selection and has mean \(\theta_j\), its conditional expectation remains \(\theta_j\).

The ordinary precision-weighted estimator, as if the interim estimate had not been selected, is \(\widetilde{\theta}_j=wX_j+(1-w)Y_j\), where \(w=v_{2j}/(v_{1j}+v_{2j})\). Conditional on retaining the arm, its expectation is

So later data reduce the contribution of interim selection bias when they receive positive weight, but do not automatically remove it. The variance-based weight \(w\) is appropriate for two unselected, unbiased estimates; by itself, it does not correct the changed conditional distribution of \(X_j\).

A Selection-Adjusted Combination

If the conditional bias \(b_j(\boldsymbol{\theta})\) is known, define the corrected interim quantity \(X_{cj}=X_j-b_j(\boldsymbol{\theta})\). Conditional on selection, it has mean \(\theta_j\). Let \(q_{1j}=\operatorname{Var}(X_j\mid S_j)\). Because subtracting a known constant does not change variance, \(X_{cj}\) also has conditional variance \(q_{1j}\). It remains independent of \(Y_j\) conditional on selection under the stated stage-independence model.

Among linear combinations of these two conditionally unbiased quantities, the minimum-variance combination gives each quantity weight proportional to its precision. In particular, if the combination is \(T=aX_{cj}+(1-a)Y_j\), its conditional variance is \(a^2q_{1j}+(1-a)^2v_{2j}\). Differentiating with respect to \(a\) gives \(2aq_{1j}-2(1-a)v_{2j}=0\), so \(a=v_{2j}/(q_{1j}+v_{2j})\). Thus

This is an oracle result: its exact conditional unbiasedness assumes the true selection bias is known. In practice, that bias usually depends on unknown treatment effects and may be estimated by a correction or model-based procedure. Replacing it with a plug-in estimate does not automatically preserve exact unbiasedness. If the correction uses later-stage estimates, the corrected interim quantity and the later estimate will generally be dependent through that plug-in. The covariance and uncertainty in the correction must then be included, or the resulting estimate should be described as approximate rather than exactly unbiased.

Deriving a Correction for a Simple Selection Rule

A two-arm example shows how selection changes both the interim mean and variance. Suppose \(X_A\) and \(X_B\) are independent interim estimates with means \(\theta_A,\theta_B\) and variances \(v_A,v_B\). Arm A is retained if \(X_A>X_B\). Define \(D=X_A-X_B\), \(\delta=\theta_A-\theta_B\), and \(s_D=\sqrt{v_A+v_B}\). Then \(D\sim N(\delta,s_D^2)\), and retaining A is the event \(D>0\).

For a normal variable \(D\) truncated below at zero, set \(a=-\delta/s_D\) and \(\lambda(a)=\phi(a)/\{1-\Phi(a)\}\), where \(\phi\) and \(\Phi\) are the standard normal density and distribution functions. Its conditional mean is \(E(D\mid D>0)=\delta+s_D\lambda(a)\). The conditional mean of \(X_A\) follows from the normal regression identity \(E(X_A\mid D)=\theta_A+(v_A/s_D^2)(D-\delta)\). Taking expectations conditional on \(D>0\) gives

The selection bias is therefore \(b_A=(v_A/s_D)\lambda(a)\). The conditional variance follows from the same normal regression decomposition: the part of \(X_A\) uncorrelated with \(D\) retains variance \(v_A-v_A^2/s_D^2\), while the conditional variance contributed by \(D\) is \((v_A^2/s_D^4)\operatorname{Var}(D\mid D>0)\). For this truncated normal, \(\operatorname{Var}(D\mid D>0)=s_D^2\{1-\lambda(a)(\lambda(a)-a)\}\). Hence

Diagnostics, Alternatives, and Limitations

Before combining stages, check that the later-stage estimate is genuinely independent of the selection decision and targets the same effect. Confirm that the variance used for each stage is appropriate, and inspect the stage-specific estimates for patterns inconsistent with a stable effect. A difference between \(X_j\) and \(Y_j\) is not by itself proof of a time trend: interim selection makes large \(X_j\) values more likely among retained arms. Conversely, a large discrepancy can flag changing populations, calendar-time effects, protocol changes, or model misspecification that a selection correction will not fix.

The main alternatives answer different questions. Using later-stage data alone is free of interim selection bias under the model, but may discard substantial information and have high variance. Naive pooling is simple and can be efficient in an unselected analysis, but generally remains conditionally biased after selection. A selection-adjusted combination uses both stages but depends on a correctly specified selection rule, a credible model for the interim estimates, and a correction whose uncertainty is addressed. Data carving uses information in the selection data through a different decomposition; it does not remove the need to model how selection affects the information being used.

In a MAMS trial, the selection event may involve several correlated arm estimates, shared controls, multiple retained arms, or more than one interim decision. Then the bias and conditional variance for one retained arm depend on the full selection rule and covariance structure, not just on a single univariate threshold. Treating armwise corrections as independent can therefore be wrong. This tutorial’s two-arm derivation is a transparent special case, not a universal correction formula.

A further limitation is that precision weighting cannot repair a target mismatch. If the treatment effect varies by stage, a pooled number may estimate a weighted average of stage-specific effects rather than a common effect. Likewise, a plug-in correction can add uncertainty and dependence that are absent from the oracle formulas. Report the estimand, selection rule, correction, and variance assumptions clearly; do not present an approximate corrected estimate as if it had exact conditional coverage or unbiasedness.