Simultaneous Confidence Intervals For All Candidate Arms

Why Interval Coverage Must Include Every Candidate Arm

In a multiarm trial, investigators may want to estimate the effect of each candidate treatment relative to a common control. If they later choose an arm using the observed results, an interval that was designed for just one prespecified comparison may not provide reliable coverage for the chosen effect. The relevant target here is a simultaneous confidence family: a collection of intervals that covers all candidate treatment effects together with a stated probability.

This familywise guarantee is useful because the intervals can be examined alongside a selection rule without first pretending that the selected arm was specified in advance. It does not, however, turn an ordinary interval into an interval with a guaranteed coverage probability conditional on which arm won. That distinction is central to interpreting the results.

Definition: Let $\theta_k$ be the effect of candidate treatment $k$ versus control, for $k=1,\ldots,K$. Intervals $I_1,\ldots,I_K$ have simultaneous coverage of at least $1-\alpha$ if $$ P\bigl(\theta_k\in I_k\text{ for every }k=1,\ldots,K\bigr)\geq 1-\alpha. $$ The probability refers to repeated trials under the stated model and design, with the effects fixed.

For a continuous outcome, one common target is the difference in population means, $\theta_k=\mu_k-\mu_0$, measured in the outcome’s units. A positive value then means a higher population mean under treatment $k$ than under control; whether that is beneficial depends on how the outcome is defined. The intervals below are for these treatment-versus-control effects, not for every possible contrast among treatments.

Bonferroni Simultaneous Intervals

A broadly applicable construction starts with a valid marginal two-sided interval for each effect. Allocate a total error probability $\alpha$ across the $K$ comparisons by using error probability $\alpha/K$ for each one. The union bound ensures that the chance of at least one interval missing its effect is no greater than the sum of the individual miss probabilities. This argument does not require the treatment-effect estimates to be independent.

Result: If each marginal interval $I_k$ satisfies $P(\theta_k\notin I_k)\leq\alpha/K$, then $$ P\bigl(\theta_k\in I_k\text{ for all }k\bigr) \geq 1-\sum_{k=1}^K P(\theta_k\notin I_k) \geq 1-\alpha. $$

For independent randomized groups with normally distributed outcomes and known variances, estimate $\theta_k$ by $\hat{\theta}_k=\bar{x}_k-\bar{x}_0$. If the treatment and control group variances are $\sigma_k^2$ and $\sigma_0^2$, and their sample sizes are $n_k$ and $n_0$, then $SE_k=\sqrt{\sigma_k^2/n_k+\sigma_0^2/n_0}$. A Bonferroni interval is

\hat{\theta}_k\ \pm\ z_{1-\alpha/(2K)}SE_k.

This formula assumes independent observations across randomized participants, normally distributed outcomes within each group, and known variances. Under these assumptions each standardized contrast has a standard normal distribution. If variances are estimated, a valid simultaneous procedure must use marginal intervals with the required error bounds, such as suitably calibrated $t$-based intervals; simply substituting estimated standard errors into a known-variance normal formula does not automatically justify the stated coverage.

Worked Example: Three Candidate Arms With Known Variances

Suppose a hypothetical trial has three candidate treatments and one control. Outcomes are normally distributed, observations are independent across participants, and the group variances are known. The estimated treatment-versus-control effects are 2.4, 1.1, and 3.0 points, with standard errors 0.80, 0.75, and 0.90 points. Construct 95% Bonferroni simultaneous intervals.

Here $K=3$ and $\alpha=0.05$. Each tail of each interval receives probability $\alpha/(2K)=0.05/6=0.008333$. The required standard-normal quantile is $z_{0.991667}\approx2.394$. The intervals are therefore:

\begin{aligned} \hat{\theta}_1 &: 2.4\pm 2.394(0.80)=2.4\pm1.915 =[0.485,\ 4.315]\text{ points},\\ \hat{\theta}_2 &: 1.1\pm 2.394(0.75)=1.1\pm1.796 =[-0.696,\ 2.896]\text{ points},\\ \hat{\theta}_3 &: 3.0\pm 2.394(0.90)=3.0\pm2.155 =[0.845,\ 5.155]\text{ points}. \end{aligned}

The family of three intervals has at least 95% simultaneous coverage under the stated assumptions. In this hypothetical result, the intervals for arms 1 and 3 exclude zero, while the interval for arm 2 includes zero. This is an interval statement about the effects in outcome points; it is not by itself a decision about clinical importance.

Justification and Dependence From a Shared Control

The Bonferroni guarantee follows from a set inclusion. If at least one interval misses its effect, the event is contained in the union of the $K$ individual miss events. The probability of that union is at most the sum of their probabilities, whether or not those events are dependent.

\begin{aligned} \left\{\text{at least one }I_k\text{ misses }\theta_k\right\} &=\bigcup_{k=1}^K\{\theta_k\notin I_k\},\\ P\left(\bigcup_{k=1}^K\{\theta_k\notin I_k\}\right) &\leq\sum_{k=1}^K P(\theta_k\notin I_k) \leq\alpha. \end{aligned}

With a shared control, treatment-effect estimates are generally correlated because each contains the same control mean. For example, when treatment and control groups are independent and the control variance is $\sigma_0^2$, the covariance between two distinct estimated effects is $\operatorname{Cov}(\hat{\theta}_i,\hat{\theta}_j)=\sigma_0^2/n_0$. Their positive correlation is one reason that treating all contrasts as independent is usually inappropriate. Bonferroni remains valid because its proof does not assume independence, though the bound may be conservative.

A joint-distribution method, such as a maxT or Dunnett-style calibration, can use the correlation among the contrasts to choose a critical value for the maximum absolute standardized error. With valid model and covariance assumptions, such a method can yield narrower intervals than Bonferroni while retaining the same familywise coverage target. The critical value must be calibrated for the actual joint distribution and design; the label “shared control” alone does not justify using a particular table or value.

Worked Example: Four Arms and the Bonferroni Critical Value

Consider a hypothetical four-arm trial with normally distributed outcomes, independent participants, and known group variances. The estimated effects are 1.8, 2.6, $-0.4$, and 1.1 points, with standard errors 0.60, 0.70, 0.50, and 0.80 points. For a 95% Bonferroni family, each tail has probability $0.05/(2\times4)=0.00625$, so the critical quantile is $z_{0.99375}\approx2.498$.

Multiplying this critical value by each standard error gives margins of $2.498(0.60)=1.499$, $2.498(0.70)=1.749$, $2.498(0.50)=1.249$, and $2.498(0.80)=1.998$ points. Thus the four simultaneous intervals are:

\begin{aligned} I_1&=[1.8-1.499,\ 1.8+1.499]=[0.301,\ 3.299],\\ I_2&=[2.6-1.749,\ 2.6+1.749]=[0.851,\ 4.349],\\ I_3&=[-0.4-1.249,\ -0.4+1.249]=[-1.649,\ 0.849],\\ I_4&=[1.1-1.998,\ 1.1+1.998]=[-0.898,\ 3.098]. \end{aligned}

All endpoints are in outcome points. The family has at least 95% simultaneous coverage under the assumptions, even if the contrasts share a control. This does not say that each interval individually has exactly 95% coverage: each marginal interval is calibrated at 98.75% so that the familywise miss probability is bounded by 5%.

What Changes When an Arm Is Selected?

Suppose a selection rule chooses an arm $S$ using the observed trial data, for example by choosing the largest estimated effect. If the simultaneous event $\{\theta_k\in I_k\text{ for all }k\}$ occurs, then it necessarily includes $\theta_S\in I_S$, whatever arm the rule selected. Therefore the simultaneous intervals provide at least $1-\alpha$ probability of covering every effect, and in particular at least $1-\alpha$ probability that the selected effect lies in its corresponding interval, across repeated trials.

This is an unconditional repeated-trial statement. It does not establish that the selected interval covers with probability at least $1-\alpha$ conditional on a particular arm being selected. Conditioning on selection changes the sampling distribution. Procedures designed specifically for conditional, selection-adjusted inference are a separate matter.

Worked Example: Reading the Interval for a Selected Arm

Return to the hypothetical three-arm results above. Suppose the prespecified selection rule chooses the arm with the largest observed treatment-versus-control estimate. The estimates are 2.4, 1.1, and 3.0 points, so the rule selects arm 3. Its interval from the simultaneous family is $[0.845,\ 5.155]$ points.

The estimate for arm 3 is the largest observed estimate, but its interval was constructed as part of the family covering all three arms. If the familywise coverage event occurs, arm 3’s true effect is covered, regardless of its rank among the observed estimates. The familywise guarantee makes no claim that, among trials in which arm 3 wins, this particular interval covers in at least 95% of those trials. Nor does the interval assign a probability to the fixed true effect.

Worked Example: The Single-Arm Boundary Case

Suppose a hypothetical trial has just one treatment-versus-control comparison, with estimated effect 1.2 points and known-variance standard error 0.50 points. At $\alpha=0.05$, $K=1$, so the Bonferroni critical value is $z_{1-0.05/(2\times1)}=z_{0.975}=1.96$.

1.2\pm1.96(0.50)=1.2\pm0.98=[0.22,\ 2.18]\text{ points}.

This is exactly the ordinary two-sided 95% normal interval: when $K=1$, Bonferroni makes no adjustment. For $K>1$, the Bonferroni critical value exceeds the ordinary two-sided $1-\alpha$ critical value, so intervals with the same standard errors are wider. The single-arm case is an important check against claiming that Bonferroni always widens intervals.

Diagnostics, Alternatives, and Limitations

Before interpreting a simultaneous family, check that its construction matches the analysis. Confirm that the family includes every candidate arm that could be selected, that the outcome scale and contrast direction are consistent, and that the standard errors and critical value correspond to the design. In particular, a shared control induces dependence; an analysis that assumes independent contrasts may use an unjustified joint critical value. For Bonferroni, dependence does not invalidate the guarantee, but incorrectly calibrated marginal intervals still do.

Bonferroni is attractive when a simple, transparent guarantee is needed or when the dependence structure is uncertain. Its cost is conservatism: the union bound can overstate the probability of at least one miss, especially when comparisons are strongly correlated. A joint maxT or Dunnett-type procedure can improve precision by using that correlation, but relies on correctly specified joint distributions and covariance estimates. Neither method removes the need to define the family of arms in advance or to account for adaptations that change the data or analysis in ways not represented by the calibration.

Common pitfall: “We selected the arm with the largest estimate, so its ordinary 95% interval is a 95% interval for the winner.” The selection rule makes the winning estimate’s distribution different from that of a prespecified estimate. Use a simultaneous family if the goal is a familywise guarantee across candidates, and do not describe that guarantee as conditional coverage for the winner.

Check Your Understanding

Answer each question, then compare with the solutions below.

For $K=5$ treatment-versus-control effects and $\alpha=0.05$, what error probability is assigned to each tail of a Bonferroni interval?
Why can Bonferroni intervals remain valid when all treatment effects use the same control group?
What does at least 95% simultaneous coverage mean if an arm is selected using the observed estimates?
For one comparison and $\alpha=0.05$, does Bonferroni change the usual two-sided normal critical value?
Name one reason a joint maxT or Dunnett-style interval may be narrower than a Bonferroni interval, and one condition it must satisfy.

Solutions. (1) Each tail receives $0.05/(2\times5)=0.005$, so the total error allocated to each comparison is 0.01. (2) The union bound applies whether or not the individual miss events are dependent; the shared control changes dependence but does not invalidate the bound. (3) Across repeated trials, the probability that all candidate effects are covered is at least 95%, which guarantees coverage for whichever arm is selected on that trial; it does not guarantee 95% coverage conditional on a particular arm winning. (4) No. With $K=1$, the critical value is $z_{0.975}=1.96$, the usual two-sided 95% value. (5) It can use the correlations among contrasts rather than the potentially conservative union bound. Its joint critical value must be calibrated for the design’s correct joint distribution and covariance structure.