Simultaneous Intervals for Several Selected Arms in MAMS Trials

Why several selected arms require joint intervals

A MAMS trial may retain more than one experimental arm after interim decisions. Reporting an interval for each retained arm is not automatically the same as giving a confidence statement for all of them together. If each interval has 95% marginal coverage, the probability that every interval covers can be less than 95%. In addition, estimates that share a control group are correlated. Finally, when the same outcomes both determine which arms are retained and estimate their effects, selection can make ordinary intervals too narrow.

The target here is a simultaneous confidence statement for the effects of the retained arms. There are two distinct adjustments to keep straight: joint calibration across arms and calibration for the selection procedure. A joint procedure addresses the first; it addresses the second only if its distribution or data source also accounts for how the arms were selected.

Definition: Let $S$ denote the observed selection outcome, including the retained arms and any relevant stopping decisions. A joint confidence region for the retained effects has conditional coverage at least $1-\alpha$ if, for every parameter vector compatible with the observed selection outcome, $$ P_{\boldsymbol{\theta}}\!\left(\boldsymbol{\theta}_{S}\in C(Y)\mid S\right)\geq 1-\alpha. $$ Here $\boldsymbol{\theta}_{S}$ is the vector of treatment effects for the retained arms and $C(Y)$ is a region computed from the trial data $Y$. Projecting $C(Y)$ onto each retained effect gives intervals whose simultaneous coverage is at least that of the region.

Shared controls and the covariance matrix

Suppose $K$ treatment arms and one control arm have independent normally distributed outcomes with common variance $\sigma^2$. For arm $j$, estimate the treatment effect by the difference between the treatment and control sample means, $\hat{\theta}_j=\bar{X}_j-\bar{X}_0$. If arm $j$ has $n_j$ participants and the shared control has $n_0$, then $$ \operatorname{Var}(\hat{\theta}_j)=\sigma^2\left(\frac{1}{n_j}+\frac{1}{n_0}\right). $$ For two distinct treatment arms $j$ and $k$, the treatment-group sample means are independent, but both estimates subtract the same control mean. Therefore, $$ \operatorname{Cov}(\hat{\theta}_j,\hat{\theta}_k) =\operatorname{Var}(\bar{X}_0) =\frac{\sigma^2}{n_0}. $$ This positive covariance must appear in the joint analysis. Treating the estimates as independent changes the shape and calibration of a multivariate confidence region.

Worked Example: Calculating Shared-Control Dependence

Suppose two hypothetical treatment arms and a control each have 48 participants. Assume independent normal outcomes with a known common standard deviation of 10 units. Calculate the standard error of either treatment-control difference and the correlation between the two estimated differences.

For either arm, the variance is $100(1/48+1/48)=200/48=4.1667$, so $$ SE=\sqrt{4.1667}=2.041\text{ units}. $$ The covariance between the two differences is the variance of the shared control mean: $$ \operatorname{Cov}(\hat{\theta}_1,\hat{\theta}_2)=\frac{100}{48}=2.083\text{ units}^2. $$ Consequently, their correlation is $2.083/\sqrt{4.1667\times4.1667}=0.500$. The estimates are positively correlated because a low observed control mean raises both estimated treatment effects, while a high control mean lowers both.

Two routes to selection-aware simultaneous inference

When the estimates used for inference are the same data that drove selection, a principled route is to condition on the observed selection outcome. For a candidate effect vector $\boldsymbol{\theta}_{S}$, use the distribution of the trial statistics under that candidate given the actual selection event. A joint test with conditional type I error no greater than $\alpha$ can then be inverted: retain all candidate vectors not rejected by the test. The resulting region has conditional coverage at least $1-\alpha$, provided the outcome model and the selection event are correctly specified.

The selection event must be represented faithfully. It can include interim efficacy or futility boundaries, which arms continued, and which arms were retained at the end. In a multi-stage trial, the relevant joint distribution must also reflect dependence across stages when participants or control information are shared. Conditioning on only the final retained set when the stopping path contains additional information may not give the intended conditional guarantee.

A second route uses estimation data independent of the selection data. For example, a prespecified validation stage can estimate the effects of arms selected using earlier-stage outcomes, provided the validation outcomes are not reused to choose the arms and the validation analysis follows the planned sampling scheme. Conditional on the selection-stage data, the retained set is fixed, and the validation estimates retain their planned distribution. Joint calibration using their covariance matrix then gives coverage conditional on selection. This route can be simpler, but it requires adequate independent information and does not make earlier-stage estimates selection-free.

Key result: If a joint region $C(Y)$ has conditional coverage at least $1-\alpha$, then the coordinate projection intervals $$ I_j(Y)=\left[\inf_{\boldsymbol{u}\in C(Y)}u_j,\ \sup_{\boldsymbol{u}\in C(Y)}u_j\right] $$ cover all retained effects simultaneously with conditional probability at least $1-\alpha$. The event that every coordinate is in its projection interval contains the event that the full effect vector is in $C(Y)$.

Projection of a multivariate confidence region

For clarity, consider $m$ retained effects with an estimator vector $\hat{\boldsymbol{\theta}}$ that is multivariate normal with mean $\boldsymbol{\theta}$ and known covariance matrix $\Sigma$. If the data used for estimation are independent of the selection data, this normal distribution applies conditional on the retained set, under the stated model. A $1-\alpha$ ellipsoidal region is $$ C(Y)=\left\{\boldsymbol{\theta}: (\hat{\boldsymbol{\theta}}-\boldsymbol{\theta})^\mathsf{T} \Sigma^{-1}(\hat{\boldsymbol{\theta}}-\boldsymbol{\theta}) \leq \chi^2_{m,1-\alpha}\right\}. $$ The quadratic form has a chi-squared distribution with $m$ degrees of freedom under the model. For coordinate $j$, the projection of this ellipsoid has half-width $$ \sqrt{\chi^2_{m,1-\alpha}\,\Sigma_{jj}}. $$ This follows by maximizing or minimizing the $j$th coordinate over the ellipsoid; correlations affect the ellipsoid’s orientation, while its coordinate projection width depends on the corresponding marginal variance. Inverting conditional tests after selection can produce a different, selection-specific region; an ordinary chi-squared ellipsoid is not automatically valid for estimates selected using the same data.

Worked Example: Projecting a Joint Region After Independent Selection

In a hypothetical trial, an interim stage selects two arms. An independent validation stage gives estimated effects of 8 and 5 units, each with standard error 2.041 units. Assume the validation estimates are jointly normal with known covariance matrix $$ \Sigma= \begin{pmatrix} 4.1667 & 2.0833\\ 2.0833 & 4.1667 \end{pmatrix}. $$ The shared-control covariance is included, and the validation-stage estimates are independent of the interim selection data. Find projected intervals from a 95% joint ellipsoid.

There are $m=2$ effects, so the 95th percentile of a chi-squared distribution with 2 degrees of freedom is 5.991. Each diagonal element of $\Sigma$ is $4.1667$, giving the common projection half-width $$ \sqrt{5.991}\times 2.041=2.448\times2.041=4.996\text{ units}. $$ Thus the projected intervals are $8\pm4.996=[3.004,12.996]$ units for the first retained arm and $5\pm4.996=[0.004,9.996]$ units for the second. Because the region is calibrated jointly and validation is independent of selection, these projected intervals cover both validation-stage effects simultaneously with probability at least 95%, conditional on the interim selection outcome. The projection intervals are wider than marginal 95% intervals because they are projections of a joint 95% region.

Worked Example: Bonferroni Intervals for Three Retained Arms

Suppose a separate hypothetical trial retains three arms using interim data and estimates their effects on an independent validation sample. The estimates are 8, 5, and 3 units, each with standard error 2.041 units. Assume the validation estimates are normal with the stated standard errors. Construct conservative simultaneous 95% intervals without needing the estimates’ correlations.

Allocate a total error probability of 0.05 equally across the three two-sided intervals. Each interval therefore has marginal confidence level $1-0.05/3=0.9833$, with critical value $z_{1-0.05/(2\times3)}\approx2.394$. The half-width is $$ 2.394\times2.041=4.886\text{ units}. $$ The intervals are $8\pm4.886=[3.114,12.886]$, $5\pm4.886=[0.114,9.886]$, and $3\pm4.886=[-1.886,7.886]$ units. By the Bonferroni inequality, the probability that all three intervals cover is at least 95%, conditional on interim selection. The guarantee does not require independence between validation estimates, though it may be conservative when their dependence is strong.

Derivation of the projection guarantee

Let the joint region satisfy $P(\boldsymbol{\theta}_{S}\in C(Y)\mid S)\geq1-\alpha$. If the full vector belongs to $C(Y)$, each coordinate must lie between the minimum and maximum values of that coordinate over the region. Hence $$ \{\boldsymbol{\theta}_{S}\in C(Y)\} \subseteq \bigcap_{j\in S}\{\theta_j\in I_j(Y)\}. $$ Taking conditional probabilities given $S$ preserves the inequality: $$ P\!\left(\bigcap_{j\in S}\{\theta_j\in I_j(Y)\}\mid S\right) \geq P(\boldsymbol{\theta}_{S}\in C(Y)\mid S) \geq1-\alpha. $$ This proves simultaneous coverage of the projected intervals. Projection can be conservative: the rectangle formed by the intervals generally contains points outside the original region, so its coverage can exceed the region’s nominal coverage.

Choosing and checking a procedure

Approach	How selection is handled	Main trade-off
Conditional joint test inversion	Uses the distribution conditional on the observed selection event.	Directly targets conditional coverage, but requires a correct model of the joint outcomes and selection rule.
Independent validation estimates	Selection uses separate data; inference uses the independent validation data.	Simple conditional justification, but precision depends on the validation information available.
Bonferroni intervals	Can be used with selection-independent estimates or individually selection-valid intervals.	Easy to apply and does not require known correlations for the coverage bound, but can be conservative.
Ordinary marginal intervals	Usually do not account for selection or simultaneous coverage.	May be narrower, but do not support a joint 95% claim merely because each interval is labelled 95%.

Before reporting intervals, check that the covariance matrix includes shared controls and any relevant cross-stage dependence, and that its entries are on the correct scale (variances and covariances have squared outcome units). Verify that the selection event used for conditional calibration matches the actual trial rule. For validation-based inference, confirm that the outcomes used for estimation were not also used to select arms. Finally, check that the number of retained arms and degrees of freedom used in the joint calibration agree with the analysis.

A warning sign is an analysis that treats shared-control contrasts as independent, or that applies an ordinary multivariate confidence region to data selected by the same outcomes without a selection argument. Another is a report that calls several marginal intervals “simultaneous” without an adjustment or a joint coverage justification. If the normal model, covariance structure, selection rule, or validation independence is doubtful, the stated conditional guarantee may not hold; the intervals should not be presented as having that guarantee.

Common pitfall: “Each selected arm has a 95% interval, so all selected effects are jointly covered with 95% probability.” This is wrong: marginal coverage does not imply joint coverage, and selection can further distort coverage. Use a joint region or a simultaneous procedure, and ensure its calibration accounts for the selection mechanism or uses genuinely independent estimation data.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

Two treatment-control estimates use the same control mean. What source of covariance does this create?
Why do projections of a valid joint region give simultaneous intervals?
Two arms are selected using interim data and estimated using independent validation data. What justifies conditional coverage after selection?
Three individually 95% intervals are reported for three retained arms. Does this establish 95% simultaneous coverage? Explain.
What is one key model or design failure that can invalidate selection-aware intervals?

Solutions. (1) Both estimates subtract the same random control mean, so their covariance includes the variance of that mean. (2) If the full effect vector is in the joint region, every coordinate is in its projected interval; therefore the joint coverage of the projections is at least the region’s coverage. (3) Conditional on interim selection, independent validation estimates retain their prespecified sampling distribution, allowing joint calibration for the retained effects. (4) No. Marginal 95% coverage alone does not guarantee that all three intervals cover together with probability at least 95%; a simultaneous adjustment or joint region is needed. (5) Examples include omitting shared-control or cross-stage covariance, using an incorrect selection event, or reusing selection outcomes as if they were independent validation data.