Simultaneous Intervals for Several Selected Arms

Why Several Selected Arms Need Joint Intervals

Interim screening may retain more than one treatment arm for further study. Reporting a separate 95% confidence interval for each retained arm does not generally give 95% confidence that all reported intervals cover their effects. If several intervals are examined together, the relevant target is their joint coverage. Screening makes this issue especially important: the set of effects to be reported is itself determined by data.

The aim here is to construct intervals for every arm in the retained set while controlling the probability that one or more intervals miss their corresponding treatment effects. A useful route is to use confirmatory data collected after screening, so that those data are independent of the screening decision. A second, more conservative route is to protect all candidate arms at once, provided each candidate’s interval has valid marginal coverage under the trial’s adaptive design.

Definition: Let $K$ be the number of candidate treatment arms, let $\Delta_j$ be the effect for candidate arm $j$ relative to control, and let $S\subseteq\{1,\ldots,K\}$ be the set retained at interim screening. A collection of intervals $C_j$, $j\in S$, has conditional simultaneous coverage of at least $1-\alpha$ if $$ P\bigl(\Delta_j\in C_j\text{ for every }j\in S\mid S\bigr)\geq 1-\alpha $$ for every possible nonempty retained set with positive probability.

The set $S$ is random, but the effects $\Delta_1,\ldots,\Delta_K$ are fixed parameters. Conditional simultaneous coverage asks for the stated coverage level after specifying which arms were retained. This is stronger than unconditional coverage, which averages over all possible screening outcomes. If no arm is retained, there are no selected-arm intervals to report and the joint-coverage event is vacuous.

A Conditional Bonferroni Construction

Suppose the interim decision uses only stage-one data, and final estimates for retained arms use independent stage-two data. For each retained arm $j$, let $\hat{\Delta}_j$ be its stage-two treatment-control estimate and let $SE_j$ be its known standard error. Assume that, conditional on $S$, each estimate has the correct normal distribution: $$ \hat{\Delta}_j\mid S\sim N(\Delta_j,SE_j^2). $$ The standard errors may depend on the retained set—for example, because the design assigns different stage-two sample sizes to different sets—but must correctly describe the conditional sampling variance. The method does not require the estimates for different arms to be independent.

If $m=|S|$ arms are retained, construct each two-sided interval at marginal confidence level $1-\alpha/m$. For known standard errors, the interval for arm $j$ is

C_j=\left[\hat{\Delta}_j-z_{1-\alpha/(2m)}SE_j,\ \hat{\Delta}_j+z_{1-\alpha/(2m)}SE_j\right],\qquad j\in S,

where $z_q$ is the $q$ quantile of the standard normal distribution. The adjustment uses the number of arms actually retained, not automatically the original number $K$, because the stage-two data are independent of which set was retained.

Justification: For each retained arm, the normal model gives $P(\Delta_j\notin C_j\mid S)=\alpha/m$. The probability that at least one selected interval misses is at most the sum of the individual miss probabilities: $$ P\left(\bigcup_{j\in S}\{\Delta_j\notin C_j\}\mid S\right) \leq\sum_{j\in S}P(\Delta_j\notin C_j\mid S) =m\frac{\alpha}{m}=\alpha. $$ Thus the probability that all $m$ intervals cover, conditional on the retained set, is at least $1-\alpha$. This union-bound argument is valid for any dependence among the intervals.

The bound can be conservative. If the estimates have a known joint normal distribution, a less conservative common critical value can instead be obtained from the conditional distribution of the maximum absolute standardized error. In that approach, choose $c$ to satisfy $$ P\left(\max_{j\in S}\left|\frac{\hat{\Delta}_j-\Delta_j}{SE_j}\right|\leq c\ \middle|\ S\right)=1-\alpha, $$ and use $\hat{\Delta}_j\pm cSE_j$. The covariance structure must be included when calculating $c$. Bonferroni intervals avoid that calculation and guarantee at least the requested coverage without assuming independence.

Worked Example: Two Arms Retained at Screening

Hypothetical trial. A trial screens three treatments after stage one and retains arms A and B. The confirmatory stage-two estimates are independent of the screening data. Treatment effects are measured as reductions in a symptom score, in points, with positive values favoring treatment. The estimates and known standard errors are $\hat{\Delta}_A=1.8$ points with $SE_A=0.50$ points, and $\hat{\Delta}_B=1.2$ points with $SE_B=0.40$ points. Construct 95% conditional simultaneous intervals.

Here $m=2$ and $\alpha=0.05$, so the Bonferroni critical value is $z_{1-0.05/(2\cdot2)}=z_{0.9875}\approx2.241$. The two intervals are:

C_A=1.8\pm2.241(0.50)=1.8\pm1.121=[0.680,\ 2.921]\text{ points},

C_B=1.2\pm2.241(0.40)=1.2\pm0.8964=[0.304,\ 2.096]\text{ points}.

Under the stated normal model and independent-stage assumption, these intervals have at least 95% probability of covering both retained-arm effects, conditional on A and B being the selected set. Both intervals are above zero, but that conclusion concerns these intervals and their stated model; it does not establish that either treatment causes a benefit in a broader patient population beyond the trial’s estimand.

Dependence and Shared Controls

Treatment-control contrasts often share a control group, so their estimates are correlated. For example, if $\hat{\Delta}_j=\bar{Y}_j-\bar{Y}_C$ and $\hat{\Delta}_k=\bar{Y}_k-\bar{Y}_C$, independent randomized groups with a common control yield $$ \operatorname{Cov}(\hat{\Delta}_j,\hat{\Delta}_k)=\operatorname{Var}(\bar{Y}_C), $$ because the same control mean is subtracted in both contrasts. Ignoring this dependence can give an incorrect critical value in a maximum-statistic procedure. It does not invalidate the Bonferroni guarantee: the union bound used above does not require independent estimates.

Worked Example: Three Retained Arms With a Common Control

Hypothetical trial. Three arms remain after screening, and their stage-two estimates all use the same control group. Assume each contrast is normally distributed with its stated known standard error, and that stage-two data are independent of screening. The estimates, in symptom-score reduction points, are 2.0 with $SE=0.40$, 1.4 with $SE=0.50$, and 0.8 with $SE=0.30$. Construct 95% simultaneous Bonferroni intervals.

There are $m=3$ selected arms, so $z_{1-0.05/(2\cdot3)}=z_{0.991667}\approx2.394$. The margins of error are $2.394(0.40)=0.9576$, $2.394(0.50)=1.197$, and $2.394(0.30)=0.7182$ points. Therefore:

Arm 1: $2.0\pm0.9576=[1.042,\ 2.958]$ points.
Arm 2: $1.4\pm1.197=[0.203,\ 2.597]$ points.
Arm 3: $0.8\pm0.7182=[0.0818,\ 1.518]$ points.

The contrasts may be correlated because they share a control. The intervals are nevertheless jointly valid at no less than 95% conditional coverage by the Bonferroni argument. A covariance-aware maximum-statistic procedure could be narrower, but its critical value would need to use the actual joint covariance matrix rather than treating the contrasts as independent.

When Stage-One Data Also Enter the Estimate

The conditional construction above relies on independence between the screening data and the data used for the intervals. If stage-one observations both determine which arms survive and contribute to the final estimates, then the selected estimates are generally dependent on the screening event. Applying a critical value based on $m=|S|$ as though the estimates were independent of selection does not, by itself, establish conditional coverage. This remains a concern even when the unselected stage-one data were collected under randomization.

One alternative is to construct valid marginal intervals for all $K$ candidate effects under the full adaptive design and apply a Bonferroni adjustment across that original family. For two-sided intervals, use the critical value $z_{1-\alpha/(2K)}$ when the marginal normal model and standard errors are valid. If all $K$ intervals cover, then every interval in any selected subset covers. This guarantees at least $1-\alpha$ unconditional coverage for the selected set. It need not guarantee $1-\alpha$ coverage conditional on each particular screening outcome. Nor does the adjustment repair marginal intervals whose sampling distributions are already invalid because of adaptation.

Worked Example: Protecting a Selected Subset From a Full-Family Procedure

Hypothetical trial. Four candidate arms have final estimates whose marginal normal models and standard errors are valid under the trial’s adaptive design. The estimates use data that also contributed to interim screening, so a conditional stage-two argument is unavailable. A full-family Bonferroni procedure is used at $\alpha=0.05$. Two arms are selected, with estimates 1.6 and 1.1 symptom-score reduction points and standard errors of 0.50 points each.

The adjustment is based on $K=4$, giving $z_{1-0.05/(2\cdot4)}=z_{0.99375}\approx2.498$. Each margin is $2.498(0.50)=1.249$ points. The selected-arm intervals are therefore $[0.351,\ 2.849]$ points and $[-0.149,\ 2.349]$ points.

Because the full set of four marginal intervals has simultaneous coverage of at least 95%, any subset chosen from those four intervals is covered whenever the full family is covered. The guarantee is unconditional over the possible screening outcomes. It is not a claim of 95% coverage conditional on the particular pair selected.

Diagnostics, Alternatives, and Limitations

Before reporting simultaneous intervals, check how the retained set was formed and which observations enter the estimates. Confirm that the standard errors match the actual stage-two allocation or, for a full-family procedure, the full adaptive design. Check that the effect scale and direction are consistent across arms, that all retained arms are included in $m$, and that any shared-control covariance is used if a joint-normal critical value is chosen. The normal known-standard-error formula is not exact when its normal model or variance assumptions fail.

The principal approaches answer related but different questions:

Approach	Coverage target	Main advantage and caution
Bonferroni over the retained arms using independent post-screening data	Conditional joint coverage given the retained set	Simple and valid under arbitrary dependence; can be conservative.
Maximum-statistic intervals using the joint covariance	Conditional joint coverage given the retained set	Can be narrower; requires a correct joint distribution and covariance.
Bonferroni over all candidate arms with valid full-design intervals	Unconditional coverage over screening outcomes	Protects any selected subset; may be wider and does not generally provide conditional coverage.

These procedures do not automatically solve every post-selection problem. In particular, a valid marginal interval for each candidate must be justified under the analysis actually used; familywise adjustment alone does not correct bias or a misspecified variance. The independent-stage method may also be less efficient if it discards stage-one information. More specialized selection-adjusted procedures can use that information, but they require a model that accounts for the screening rule and are not obtained by simply changing the number of intervals in an ordinary confidence formula.

Common Pitfall

Wrong reasoning: “Only two arms were retained, so I can use a two-arm Bonferroni correction on the pooled estimates and claim 95% coverage conditional on selection.” The number of retained arms does not remove the dependence between pooled estimates and the data-driven screening decision.

Correction: A correction based on the observed $m$ gives the stated conditional guarantee when the inference data are independent of screening and the conditional model is valid. If screening data enter the estimates, either use a method that accounts for the selection mechanism or use valid full-family intervals and describe their guarantee accurately as unconditional.

Key takeaway: State the coverage target before choosing the critical value. Independent post-screening data support conditional simultaneous coverage over the retained set; valid intervals protected across all candidate arms support unconditional coverage for any selected subset.

Check Your Understanding

Answer each question, then compare with the solutions below.

Four arms are retained, and independent confirmatory data are used. At a 5% familywise error level, what marginal two-sided confidence level is assigned to each Bonferroni interval?
Why does Bonferroni remain valid when treatment-control contrasts share a control group?
Three candidate arms have valid full-design marginal intervals protected together at 95%. What coverage guarantee follows for the subset chosen at interim screening?
Two arms are selected, but the pooled estimates include the screening data. Is it enough to use the two-arm Bonferroni critical value to claim conditional 95% coverage? Explain.

Solutions. (1) Each interval has marginal confidence level $1-\alpha/m=1-0.05/4=0.9875$, or 98.75%; the two-sided normal critical value is $z_{0.99375}$. (2) The union bound limits the probability of at least one miss by the sum of the marginal miss probabilities, regardless of dependence. (3) Whenever all three candidate intervals cover, every selected subset is covered; this gives at least 95% unconditional coverage over screening outcomes, not necessarily conditional coverage for each selected set. (4) No. The pooled estimates are generally dependent on selection; the $m=2$ conditional argument requires independent inference data or another procedure that explicitly handles selection.