Combining Stagewise Data After Arm Selection

Why Combine Stagewise Data?

When a multi-arm, multi-stage design drops some treatments at an interim analysis, participants already observed on the retained arm still provide information about its treatment effect. Combining those data with the later-stage data can improve precision compared with using only the later stage. But the interim data helped determine which arm continued. For the retained arm, its interim estimate may therefore be unusually high or otherwise selection-dependent. Pooling it with new data does not make that dependence disappear.

The key distinction is between how data contribute to an estimate and whether those data are independent of the selection event. In the setting considered here, selection uses stage 1 data, while stage 2 uses new, independent participants. The stage 2 estimate for any fixed arm is then independent of the stage 1 selection event; the stage 1 estimate for the selected arm generally is not.

Definition: Let $\hat\theta_{jk}$ be the estimated treatment effect for arm $j$ at stage $k$, with variance $V_{jk}$. The stagewise estimates are independent across stages when they are calculated from independent stage-specific data and the design creates no shared or otherwise dependent observations across stages. If the same treatment effect $\theta_j$ applies at both stages and the variances are known, the inverse-variance combined estimate for a retained arm $j$ is $$ \hat\theta_{j,\mathrm{comb}} = \frac{\hat\theta_{j1}/V_{j1}+\hat\theta_{j2}/V_{j2}} {1/V_{j1}+1/V_{j2}}, \qquad \operatorname{Var}(\hat\theta_{j,\mathrm{comb}}) = \frac{1}{1/V_{j1}+1/V_{j2}}. $$

Assumptions Behind the Combination

The formula is not just an arithmetic rule. Its usual variance and interpretation rely on assumptions that should be checked for the design and analysis being used.

Common estimand across stages. Both stage estimates target the same effect $\theta_j$, on the same outcome scale and for a compatible population. If the treatment effect changes across stages or the estimand changes, a single common-effect estimate may not answer the intended question.
Unbiased stage estimates before selection. For a fixed arm, $E(\hat\theta_{jk})=\theta_j$ under the sampling model. This does not imply that the stage 1 estimate remains unbiased after conditioning on the event that arm $j$ was retained.
Correct variances and independence across stages. The $V_{jk}$ values describe the sampling variances, and the stage estimates use independent data. If estimates are correlated, the independent-stage formula for the variance is wrong.
Weights fixed independently of the observed estimates. The stated inverse-variance weights are appropriate when the variances and sample sizes determining them are fixed or treated as fixed. Data-dependent weights can require a separate analysis.
Selection uses only the stated interim information. To claim stage 2 is independent of selection, selection must be determined without stage 2 outcomes, and stage 2 data must be independent of stage 1 data.

For a difference in mean outcomes, for example, a stage-specific estimate could be the treatment-group sample mean minus the control-group sample mean. Its variance is the sum of the two sample-mean variances when the groups are independent. The same construction can be used at each stage, provided that the stage-specific participants are distinct and the effect and scale remain comparable.

Why Inverse-Variance Weighting Works

Suppose, for a fixed arm, the two stage estimates are independent, each has expectation $\theta_j$, and their variances $V_{j1}$ and $V_{j2}$ are known. Consider a linear combination $a\hat\theta_{j1}+(1-a)\hat\theta_{j2}$. Its coefficients sum to one, so its expectation is $\theta_j$. Independence gives its variance as $a^2V_{j1}+(1-a)^2V_{j2}$. Minimizing this variance over $a$ gives the inverse-variance weights.

\begin{aligned} \frac{d}{da}\left[a^2V_{j1}+(1-a)^2V_{j2}\right] &=2aV_{j1}-2(1-a)V_{j2}=0,\\ a&=\frac{V_{j2}}{V_{j1}+V_{j2}},\\ 1-a&=\frac{V_{j1}}{V_{j1}+V_{j2}}. \end{aligned}

These coefficients are equivalently proportional to the precisions $1/V_{jk}$. Substituting them into the variance gives $$ \operatorname{Var}(\hat\theta_{j,\mathrm{comb}}) = \frac{V_{j1}V_{j2}}{V_{j1}+V_{j2}} = \frac{1}{1/V_{j1}+1/V_{j2}}. $$

Worked Example: Combining Two Independent Stage Estimates

Consider a hypothetical trial in which arm A is retained after stage 1. Its estimated treatment effects are $5.2$ outcome units at stage 1, with standard error $2.0$, and $1.8$ outcome units at stage 2, with standard error $1.5$. Assume independent stage data, a common effect across stages, and fixed variances.

The variances are $V_{A1}=2.0^2=4.00$ and $V_{A2}=1.5^2=2.25$. Thus the inverse-variance combined estimate is

\hat\theta_{A,\mathrm{comb}} = \frac{5.2/4.00+1.8/2.25}{1/4.00+1/2.25} = \frac{1.30+0.800}{0.250+0.4444} = 3.024\text{ outcome units}.

Its model-based standard error is $$ SE(\hat\theta_{A,\mathrm{comb}}) = \sqrt{\frac{1}{0.250+0.4444}} = 1.200\text{ outcome units}. $$ The stage 2 estimate receives more weight because it has the smaller variance. This calculation combines the observed estimates; it does not establish that the combined estimate is unbiased conditional on arm A having been selected.

Independence from the Selection Event

Let $\mathcal F_1$ denote all information available at the stage 1 selection. Let $J$ be the arm retained, with $J$ determined using only $\mathcal F_1$. Suppose the stage 2 data are independent of $\mathcal F_1$, and the analysis of stage 2 for arm $j$ uses only those new data. Then, for each fixed $j$, the stage 2 estimate $\hat\theta_{j2}$ is independent of any event determined by $\mathcal F_1$, including $\{J=j\}$. In probability terms, for any such event $E$ and any set $B$,

P(\hat\theta_{j2}\in B,\ E) = P(\hat\theta_{j2}\in B)P(E).

This is the useful independence result: conditional on arm $j$ being retained, its stage 2 estimate still has its stage-specific sampling distribution, under the stated assumptions. By contrast, $\hat\theta_{j1}$ contributed to the decision to retain the arm. Conditional on $\{J=j\}$, its distribution may be shifted or otherwise altered by that decision.

The combined estimate uses both quantities. Write its stage 1 and stage 2 coefficients as $a_1=V_{j2}/(V_{j1}+V_{j2})$ and $a_2=V_{j1}/(V_{j1}+V_{j2})$. Conditional on $J=j$, its expectation is $$ E(\hat\theta_{j,\mathrm{comb}}\mid J=j) = a_1E(\hat\theta_{j1}\mid J=j)+a_2\theta_j. $$ Therefore, if the conditional bias of the stage 1 estimate is $b_{j1}=E(\hat\theta_{j1}\mid J=j)-\theta_j$, the conditional bias of the combined estimate is $a_1b_{j1}$. The independent stage 2 data dilute that conditional bias according to their weight; they do not generally erase it.

Worked Example: How Later Data Dilute Interim Selection Bias

Suppose a hypothetical arm has true effect $\theta_j=3.0$ outcome units. Its stage 1 estimate has variance $4.00$, and its stage 2 estimate has variance $1.00$. Assume the stage 2 estimate is independent of selection and has expectation $3.0$. Under a particular selection event, suppose the conditional mean of the stage 1 estimate is $5.0$ outcome units. This conditional mean is an assumption for the illustration, not a newly observed trial result.

The combined estimate gives stage 1 weight $1.00/(4.00+1.00)=0.200$ and stage 2 weight $4.00/(4.00+1.00)=0.800$. Its conditional expectation is

E(\hat\theta_{j,\mathrm{comb}}\mid J=j) = 0.200(5.0)+0.800(3.0) = 3.40\text{ outcome units}.

The stage 1 conditional bias is $5.0-3.0=2.0$ outcome units. The combined conditional bias is $3.40-3.0=0.400$ outcome units, which equals $0.200(2.0)$. The later estimate reduces the influence of the selected interim estimate, but the pooled estimate remains conditionally biased in this example.

When the Independence Argument Fails

Independence is a property of the data-generating and selection processes, not a consequence of labelling observations as different stages. It can fail if participants appear in both stages, if a control group is reused, if stage 2 outcomes influence which arm is called selected, or if stage 2 allocation or analysis uses information that makes its estimate dependent on the interim data. A selection path that includes decisions at later interim analyses also differs from a selection event based only on stage 1: an estimate used to make a later decision is not independent of that expanded path.

If the estimates are correlated, inverse-variance weighting based on independence is generally not minimum-variance, and its variance calculation is incorrect. For two unbiased estimates of the same effect with covariance matrix $\Sigma$, the minimum-variance linear unbiased combination is $$ \hat\theta = \frac{\mathbf{1}^{T}\Sigma^{-1}\hat{\boldsymbol\theta}} {\mathbf{1}^{T}\Sigma^{-1}\mathbf{1}}, \qquad \operatorname{Var}(\hat\theta) = \frac{1}{\mathbf{1}^{T}\Sigma^{-1}\mathbf{1}}, $$ where $\hat{\boldsymbol\theta}$ is the vector of stage estimates and $\mathbf{1}$ is a vector of ones. This handles covariance in the combination, but it does not by itself adjust for selection-dependent sampling distributions.

Worked Example: Correcting for Correlated Stage Estimates

Suppose, hypothetically, that two estimates of the same effect are $5.0$ and $3.0$ outcome units. Each has variance $4.00$, but they share control information and have covariance $1.00$. Their covariance matrix is

\Sigma= \begin{pmatrix} 4.00&1.00\\ 1.00&4.00 \end{pmatrix}.

Because the variances are equal, the minimum-variance unbiased combination assigns equal weights. Its estimate is $(5.0+3.0)/2=4.00$ outcome units. Its variance, including the covariance term, is $$ \operatorname{Var}(\hat\theta) = \frac{4.00+4.00+2(1.00)}{4} = 2.50, $$ so its standard error is $\sqrt{2.50}=1.581$ outcome units. Treating the estimates as independent would instead give variance $(4.00+4.00)/4=2.00$ and standard error $1.414$. That smaller standard error is not justified when the shared control creates covariance.

Comparing Estimation Choices and Checking the Analysis

The choice to combine data should be distinguished from the choice to adjust for selection. An inverse-variance combination uses information from both stages under a common-effect model. A stage-2-only estimate discards interim information and, when stage 2 is independent of selection, avoids the direct influence of the selected stage 1 estimate. It will generally be less precise. Neither choice, by itself, answers every post-selection inferential question; in particular, the usual combined standard error describes the fixed-arm sampling model and does not automatically describe the distribution conditional on the arm having been selected.

Approach	What it uses	Main consideration
Inverse-variance combination	Estimates from all included stages	Uses interim information efficiently under the common-effect and covariance assumptions, but can retain selection dependence.
Stage-2-only estimate	New data after selection	Can be independent of earlier selection, but generally has less information and greater variance.
Unweighted average	All stage estimates with equal weights	Usually not minimum-variance when stage variances differ; equal weighting does not correct selection bias.

Before reporting a combined estimate, check which participants and controls contribute at each stage, whether any data are reused, how the arm was selected, and whether later decisions are included in the selection event being discussed. Also check that the effect scale and estimand match across stages, and that the variances used for weighting are appropriate. If sample sizes, variances, or weights depend on interim outcomes, the fixed-weight derivation above may not apply without modification.

Common Pitfall

A common error is: “Stage 2 is independent of stage 1, so the combined estimate is independent of selection.” The correction is that only the stage 2 estimate for a fixed arm is independent of a selection event determined entirely by stage 1, under independent stage data. The combined estimate still contains the stage 1 estimate, which helped determine selection. Its conditional distribution can therefore depend on the selection event. Independence of one component does not transfer to a combination that includes a dependent component.

Key takeaway: Inverse-variance weighting combines stage estimates efficiently under a common-effect model and correct independence or covariance assumptions. If arm retention is based on stage 1 data, later independent stage estimates can be independent of that selection event, but the pooled estimate still carries the selection dependence of its interim component.

Check Your Understanding

Answer each question, then compare with the solutions below.

Two independent estimates of the same effect are $2.0$ units with standard error $1.0$, and $5.0$ units with standard error $2.0$. Calculate their inverse-variance combined estimate and standard error.
Selection is determined using only stage 1 data, and stage 2 data are independent of stage 1. Which estimate is independent of the event that a fixed arm was retained: its stage 1 estimate, its stage 2 estimate, or their combination?
A stage 1 estimate has conditional bias $1.5$ units after selection. Its weight in the combined estimate is $0.30$. Under the common-effect assumptions, what is the combined estimate’s conditional bias?
Why can an independent-stage standard error be too small when the same control observations contribute to both stage estimates?

Solutions. (1) The variances are $1.00$ and $4.00$, so $$ \hat\theta_{\mathrm{comb}} = \frac{2.0/1.00+5.0/4.00}{1/1.00+1/4.00} = \frac{3.25}{1.25} = 2.60\text{ units}. $$ Its standard error is $\sqrt{1/1.25}=0.8944$ units. (2) The fixed-arm stage 2 estimate is independent of the retention event if it uses independent new data and selection uses only stage 1. The stage 1 estimate is generally selection-dependent, and the combination includes it. (3) The combined conditional bias is $0.30(1.5)=0.450$ units. (4) Reusing controls can make the estimates positively correlated. The independent-stage variance formula omits the positive covariance term, understating the variance and standard error.