Estimating Contrasts Between Selected Active Arms

Why a Contrast Between Selected Arms Needs Its Own Analysis

In a multi-arm multi-stage trial, investigators may retain more than one active treatment and want to compare those treatments directly. The estimand is then the difference between their treatment effects, not either treatment effect relative to control. This distinction matters twice: the active-arm contrasts share a control estimate, and the decision to retain the arms depends on interim data that are associated with later estimates.

The shared control creates covariance even before selection is considered. Selection adds a second issue: conditional on the event that a particular pair was retained, its observed difference may no longer be centered on the population difference. A sound analysis therefore specifies both the contrast and the selection event, and accounts for their joint distribution.

Definition: Let \(\theta_i=\mu_i-\mu_0\) denote the treatment effect for active arm \(i\) relative to the common control, with larger outcomes assumed beneficial. For two retained active arms \(i\) and \(j\), the active-arm contrast is \(\delta_{ij}=\theta_i-\theta_j=\mu_i-\mu_j\). Its direct estimator is \(\hat\delta_{ij}=\hat\theta_i-\hat\theta_j\). The contrast describes the difference in mean outcome between the two active treatments; the common-control mean cancels algebraically.

Covariance From a Shared Control

For a simple calculation, suppose the groups are independent, observations within each group are independent with finite variances, and the sample means are unbiased for their group means. Write \(v_g=\sigma_g^2/n_g\) for the variance of the mean in group \(g\). Then \(\hat\theta_i=\bar X_i-\bar X_0\), so \( \operatorname{Var}(\hat\theta_i)=v_i+v_0 \). For two different active arms, the only random component they share is \(\bar X_0\); consequently \( \operatorname{Cov}(\hat\theta_i,\hat\theta_j)=v_0 \). Subtracting the estimates removes that shared component: \( \hat\delta_{ij}=\bar X_i-\bar X_j \), with variance \(v_i+v_j\).

These identities assume independent treatment-group data and a common control mean used in both contrasts. In an actual MAMS design, repeated looks can reuse data, and allocation, covariate adjustment, or other design features can change the covariance structure. The relevant covariance must then come from the joint model for the estimates at the selection and estimation stages, rather than from an independence shortcut.

Worked Example: Two Treatment Effects Sharing One Control

Suppose a hypothetical parallel-group trial has independent normally distributed outcomes with known standard deviation 4 points in all groups. There are 40 participants in the control group and 40 in each of active arms A and B. The observed means are 10.0, 13.2, and 12.0 points, respectively. First ignore selection and calculate the direct contrast and its standard error.

The estimated effects relative to control are \( \hat\theta_A=13.2-10.0=3.2 \) points and \( \hat\theta_B=12.0-10.0=2.0 \) points. Thus \( \hat\delta_{AB}=3.2-2.0=1.2 \) points, which is also \(13.2-12.0\). Each group mean has variance \(4^2/40=0.4\) square points. Therefore \( \operatorname{Var}(\hat\theta_A)=0.4+0.4=0.8 \) and \( \operatorname{Var}(\hat\theta_B)=0.8 \), while \( \operatorname{Cov}(\hat\theta_A,\hat\theta_B)=0.4 \).

The contrast variance is \(0.8+0.8-2(0.4)=0.8\) square points, so \(SE(\hat\delta_{AB})=\sqrt{0.8}=0.8944\) points. Equivalently, its variance is the sum of the active-arm mean variances, \(0.4+0.4=0.8\). If the shared-control covariance were incorrectly set to zero, the calculated variance would be \(1.6\), and the standard error would be \(\sqrt{1.6}=1.265\) points. That calculation is too large for this direct contrast because it counts the shared control contribution twice rather than letting it cancel.

Joint Selection Changes the Contrast Distribution

The preceding calculation describes the unconditional sampling variance under its stated model. If a pair is chosen using interim results, the estimates for that pair are observed conditional on a selection event. A correction based only on the two marginal treatment effects does not generally give the conditional distribution of their difference: selection can depend on both estimates, their covariance, their ranking, and the other arms that could have been retained.

A useful general representation separates the final contrast from the data used for selection. Let \(D\) be the final estimate of a specified active-arm contrast, and let \(U\) be a vector of interim estimates or statistics that determine whether that pair is retained. Suppose their joint distribution is multivariate normal with means \(\delta\) and \(\mu_U\), and covariance blocks \(\operatorname{Var}(D)=V_D\), \(\operatorname{Var}(U)=V_U\), and \(\operatorname{Cov}(D,U)=C\). Let \(A\) be the event that the specified pair is retained under the trial's selection rule. If \(V_U\) is nonsingular, normal regression gives the conditional mean of \(D\) given \(U\); averaging that expression over \(U\) conditional on \(A\) gives the selection shift.

E(D\mid A) = \delta+C V_U^{-1}\{E(U\mid A)-\mu_U\}.

The expression is a mean shift, not an automatic plug-in correction. It depends on the selection region, the model parameters, and the joint covariance. If those quantities were known, subtracting the shift from the observed \(D\) would remove its conditional mean displacement for that event. In practice, parameters are unknown, so a selection-aware method must estimate or integrate over them; a correction evaluated at estimated parameters is not guaranteed to be exactly conditionally unbiased.

Deriving the Selection Shift

Under the joint normal model, the residual \(R=D-\delta-CV_U^{-1}(U-\mu_U)\) has mean zero and is independent of \(U\). Independence follows because its covariance with \(U\) is \(C-CV_U^{-1}V_U=0\), and zero covariance implies independence for jointly normal variables. Since the event \(A\) is determined by \(U\), the residual also has conditional mean zero given \(A\). Taking conditional expectations in the definition of \(R\) proves the displayed result.

\begin{aligned} E(D\mid A) &=\delta+CV_U^{-1}\{E(U\mid A)-\mu_U\}+E(R\mid A)\\ &=\delta+CV_U^{-1}\{E(U\mid A)-\mu_U\}. \end{aligned}

This derivation also explains why covariance is central. If the final contrast is uncorrelated with every selection statistic, then \(C=0\) and this particular conditional mean shift is zero. If it is correlated with the selection statistics, truncating or ranking those statistics generally changes the conditional mean. The condition \(C=0\) does not by itself establish that every aspect of inference is unaffected; it only removes this mean-shift term under the stated model.

Worked Example: Correcting for a Selection Ranking Component

Consider a hypothetical situation in which two active arms have been retained, and the interim estimate \(W\) of their difference is used to label the arm with the larger interim estimate as the apparent winner. Focus on the ranking component of selection, \(A=\{W>0\}\), where positive values favor arm 1 over arm 2. Suppose \(W\) is normal with mean \(\delta=0.4\) points and standard deviation \(0.5\) points. The final contrast \(D\) has mean \(\delta\), standard deviation \(0.4\) points, and correlation \(0.6\) with \(W\). This example isolates a one-dimensional selection component; it does not represent every region of a full multi-arm retention rule.

For a normal variable truncated below at zero, let \(a=(0-\delta)/0.5=-0.8\). Then \(E(W\mid W>0)=\delta+0.5\,\phi(a)/\{1-\Phi(a)\}\), where \(\phi\) and \(\Phi\) are the standard normal density and distribution functions. Using \(\phi(-0.8)=0.2897\) and \(1-\Phi(-0.8)=0.7881\), the truncated mean is \(0.4+0.5(0.2897/0.7881)=0.5838\) points. Its displacement from the unselected mean is \(0.1838\) points.

The covariance between \(D\) and \(W\) is \(0.6(0.4)(0.5)=0.12\) square points, and \(\operatorname{Var}(W)=0.25\). The conditional mean shift in \(D\) is therefore \((0.12/0.25)(0.1838)=0.0882\) points. Thus \(E(D\mid W>0)=0.4+0.0882=0.4882\) points. With these stipulated parameters, subtracting \(0.0882\) points from a final estimate would remove this event's conditional mean shift. In an analysis with unknown parameters, that shift must itself be estimated, and the calculation does not account for any additional pair-retention conditions.

Contrasts Across Several Active Arms

For several arms, it is useful to express contrasts as linear combinations. Let \(\hat{\boldsymbol\theta}\) be a vector of estimated treatment effects against the common control, with covariance matrix \(\Sigma\). A vector of contrasts is \(B\hat{\boldsymbol\theta}\), where each row of \(B\) specifies one contrast. Its covariance matrix is \(B\Sigma B^\mathsf{T}\). This matrix includes both the common-control covariance and any covariance induced by shared observations across stages.

Worked Example: Covariance Between Two Selected-Pair Contrasts

Suppose three active arms A, B, and C are compared with one control in a hypothetical independent-groups trial. The control has \(n_0=50\) and known outcome variance 25 square points. The active-arm sample sizes are 40 each, with known variances 36, 16, and 10 square points for A, B, and C. The observed means are 20, 24, 22, and 23 points for control, A, B, and C. Calculate the two contrasts A versus B and A versus C, and their covariance.

The estimated treatment effects are \(4\), \(2\), and \(3\) points. Thus \(\hat\delta_{AB}=4-2=2\) points and \(\hat\delta_{AC}=4-3=1\) point. Their direct forms are \(\bar X_A-\bar X_B\) and \(\bar X_A-\bar X_C\). The variance of the A mean is \(36/40=0.9\), that of B is \(16/40=0.4\), and that of C is \(10/40=0.25\), all in square points.

Consequently, \(\operatorname{Var}(\hat\delta_{AB})=0.9+0.4=1.3\) and \(\operatorname{Var}(\hat\delta_{AC})=0.9+0.25=1.15\). The contrasts share the A mean, so \(\operatorname{Cov}(\hat\delta_{AB},\hat\delta_{AC})=\operatorname{Var}(\bar X_A)=0.9\) square points. The control contribution cancels from both contrasts, but the A contribution is shared and does not cancel. The resulting correlation is \(0.9/\sqrt{1.3(1.15)}=0.736\) to three significant figures. If selection chose these contrasts using the same interim data, this substantial dependence would need to be represented in the joint selection model.

Choosing and Checking an Analysis

A direct, unadjusted contrast is the natural estimator when the pair was specified independently of the data used to estimate it, or when the analysis explicitly targets an unconditional sampling distribution. When the pair is selected adaptively, a selection-aware analysis instead conditions on, or otherwise models, the rule that generated the observed pair. The conditional-normal calculation above gives one route when the joint normal model and selection region are adequately specified. Other selection-aware approaches can use a conditional likelihood, test inversion, or simulation under the full design; their validity still depends on representing the selection process and covariance correctly.

Subtracting two separately corrected arm estimates is not generally equivalent to correcting the selected contrast directly. The marginal adjustment for each arm can condition on a different event or use different marginal information. The desired contrast is selected jointly, and its correction depends on its covariance with the full vector of selection statistics. Simultaneous intervals for multiple selected arms address a related but distinct inferential goal; they do not automatically supply a selection-adjusted estimate of a particular active-versus-active difference.

Before interpreting a correction, check that the stated selection rule matches the trial: which looks and outcomes were used, what thresholds or ranking rules applied, and how ties or early stopping were handled. Check whether the covariance matrix is positive definite or whether a justified lower-dimensional representation is needed; check that its entries reflect shared controls and reused data; and examine whether the assumed normal approximation is plausible at the relevant sample sizes. A material mismatch between the actual adaptive rule and the modeled event can invalidate the calculated shift.

Ignoring shared-control covariance: treating treatment-versus-control estimates as independent gives the wrong variance for their difference.
Correcting each arm separately: marginal corrections need not equal a correction conditioned on the joint event that selected the pair.
Using the wrong selection event: conditioning only on a ranking when retention also depended on thresholds, other arms, or earlier looks can omit important selection information.
Overstating a plug-in correction: a shift calculated using estimated parameters is an approximation unless the method establishes the desired calibration.

The method also has limits. The normal conditional-mean formula requires a joint normal model for \(D\) and \(U\), and the event must be representable in terms of the modeled selection variables. Model misspecification, sparse data, complex adaptive rules, or uncertain covariance estimates can make the correction unstable. Even with a correctly calculated mean shift, estimation and interval uncertainty must reflect the selection procedure; subtracting a shift alone does not validate an ordinary unadjusted interval.

Key takeaway: For selected active arms, define the direct contrast, construct its covariance with the full selection vector, and condition on the actual joint selection event. The common control cancels from a direct active-versus-active contrast, but it still affects the dependence of treatment-effect estimates and must be handled consistently.

Check Your Understanding

Answer each question, then compare with the solutions.

Two independent active-arm means have variances 0.6 and 0.4 square points and share a control mean with variance 0.3 square points. What is the covariance between their treatment-effect estimates, and what is the variance of their direct contrast?
In the joint normal selection formula, what happens to the conditional mean shift if the final contrast is uncorrelated with every selection statistic?
Why may subtracting two marginally selection-corrected treatment estimates fail to correct the contrast between the selected arms?
Give two features of a MAMS design that should be represented when constructing the covariance matrix for a selection-aware contrast.

Solutions. (1) The shared control gives covariance \(0.3\) square points. The direct contrast variance is \(0.6+0.4=1.0\) square points; equivalently, \((0.6+0.3)+(0.4+0.3)-2(0.3)=1.0\). (2) If \(C=0\), then \(CV_U^{-1}\{E(U\mid A)-\mu_U\}=0\), so the formula gives no conditional mean shift for that contrast under the model. (3) The marginal corrections may condition on different information, whereas the contrast correction depends on the joint event selecting the pair and on the contrast's covariance with the selection vector. (4) Examples include the covariance induced by a shared control and the correlation between interim and final estimates when observations are reused across looks.