Simultaneous Confidence Intervals From FWER Control

Why Simultaneous Intervals Are Different

A multi-arm trial may estimate several treatment effects, such as the effects of three experimental arms compared with a control. Reporting an interval for each effect is useful, but three separate 95% confidence intervals do not generally provide 95% confidence that all three effects are covered together. That joint claim is stronger than a collection of individual claims.

Familywise error rate (FWER) control addresses the related testing problem: across a prespecified family of hypotheses, the probability of making one or more false rejections is at most $\alpha$. Inverting such tests gives confidence sets whose joint coverage is at least $1-\alpha$. The family must be defined before interpreting the results; here, assume it consists of $m$ treatment effects $\theta_1,\ldots,\theta_m$.

Definition: A collection of confidence sets $C_1,\ldots,C_m$ has simultaneous coverage of at least $1-\alpha$ if $$ P(\theta_1\in C_1,\ldots,\theta_m\in C_m)\geq 1-\alpha. $$ The probability refers to the repeated-sampling behaviour of the procedure. It does not mean that, after observing the data, there is probability $1-\alpha$ that the fixed effects lie in the reported intervals.

Inverting Familywise Tests

For a candidate parameter vector $(\theta_1^0,\ldots,\theta_m^0)$, consider testing the family of component null hypotheses $H_i:\theta_i=\theta_i^0$. Apply a procedure that controls FWER at level $\alpha$, using valid component tests for the hypotheses in this family. The candidate vector belongs to the inverted confidence set if the procedure rejects none of its component hypotheses.

This gives a joint confidence set: collect every candidate vector for which no component is rejected. To see why it has the advertised coverage, evaluate the procedure at the true vector. Any exclusion of the true vector requires rejecting at least one true component null. FWER control bounds the probability of that event by $\alpha$. Projecting the joint set onto each effect gives marginal sets; taking their interval hulls, when needed, gives simultaneous intervals that contain the joint set. Such projections can be conservative or difficult to compute.

Theorem: If the testing procedure controls FWER at level $\alpha$ for every configuration in the prespecified family, inverting it gives a joint confidence set with coverage at least $1-\alpha$. Any coordinatewise intervals that contain the corresponding projections of that joint set also have simultaneous coverage of at least $1-\alpha$.

For routine reporting, a simpler construction is often available: invert a valid test for each effect at a component level $\alpha_i$, where the allocations satisfy $\sum_{i=1}^m\alpha_i\leq\alpha$. This is the confidence-interval counterpart of Bonferroni testing. It yields a rectangular collection of intervals directly, without needing to calculate projections of a joint region.

Derivation: Let $E_i$ be the event that interval $i$ fails to cover $\theta_i$. Then $$ P(\text{at least one interval misses}) =P\left(\bigcup_{i=1}^m E_i\right) \leq \sum_{i=1}^m P(E_i) \leq \sum_{i=1}^m\alpha_i \leq\alpha. $$ Therefore $P(\theta_i\in C_i\text{ for every }i)\geq1-\alpha$. The union bound does not require independence among the treatment-effect estimates.

Bonferroni Intervals for Normal Estimates

Suppose each treatment effect has an estimate $\hat{\theta}_i$ with standard error $SE_i$, and the standardized estimate has a standard normal distribution under the corresponding null hypothesis. Assume the standard errors and the normal approximation are appropriate for the trial analysis, and that each interval uses a valid marginal procedure. For two-sided intervals, assigning level $\alpha_i$ to effect $i$ gives

C_i= \left[ \hat{\theta}_i-z_{1-\alpha_i/2}SE_i,\; \hat{\theta}_i+z_{1-\alpha_i/2}SE_i \right], \qquad \sum_{i=1}^m\alpha_i\leq\alpha,

where $z_q$ is the $q$-quantile of the standard normal distribution. Equal allocation sets $\alpha_i=\alpha/m$. These intervals may be wider than pointwise intervals because each component test uses less than the full level $\alpha$.

Worked Example: Three Two-Sided Treatment-Effect Intervals

Suppose a hypothetical trial estimates three treatment effects, each measured in points on a symptom scale. The estimates and standard errors are $(4.2,1.5)$, $(2.8,1.2)$, and $(1.1,1.0)$, respectively, where each pair is $(\hat{\theta}_i,SE_i)$. Assume the normal approximation and standard errors are valid. For a familywise level of $\alpha=0.05$, use equal Bonferroni allocation.

Each component receives $\alpha_i=0.05/3=0.01667$. The two-sided critical value is $z_{1-0.01667/2}=z_{0.99167}\approx2.394$. Thus each interval is $\hat{\theta}_i\pm2.394SE_i$.

For the first effect, $4.2\pm2.394(1.5)=4.2\pm3.591$, giving $(0.609,7.791)$ points. For the second, $2.8\pm2.394(1.2)=2.8\pm2.873$, giving $(-0.073,5.673)$ points. For the third, $1.1\pm2.394(1.0)=1.1\pm2.394$, giving $(-1.294,3.494)$ points.

Under the stated assumptions, the probability that all three intervals cover their respective effects is at least 0.95. The second and third intervals include zero, so the data do not rule out a zero effect for those comparisons using this simultaneous procedure. The first interval is entirely above zero.

Unequal Error Allocation and One-Sided Bounds

Equal allocation is not required. Weighted Bonferroni intervals assign larger $\alpha_i$ to comparisons given greater priority, while preserving $\sum_i\alpha_i\leq\alpha$. A larger allocation gives a smaller critical value and generally a narrower interval. This trade-off must be specified as part of the analysis, not chosen after seeing which results are favourable.

Worked Example: Unequal Allocation Across Three Effects

Suppose a hypothetical study estimates effects in points as $\hat{\theta}=(3.0,1.4,-0.5)$, with standard errors $SE=(0.9,0.8,0.7)$. Assume valid normal inference. Allocate familywise $\alpha=0.05$ as $(\alpha_1,\alpha_2,\alpha_3)=(0.025,0.015,0.010)$. The allocations sum to 0.05, so the Bonferroni guarantee applies.

The two-sided critical values are approximately 2.241, 2.432, and 2.576. The first interval is $3.0\pm2.241(0.9)=3.0\pm2.017$, or $(0.983,5.017)$ points. The second is $1.4\pm2.432(0.8)=1.4\pm1.946$, or $(-0.546,3.346)$ points. The third is $-0.5\pm2.576(0.7)=-0.5\pm1.803$, or $(-2.303,1.303)$ points.

The first comparison received the largest error allocation and therefore has the narrowest interval relative to its standard error. This does not make it more informative by itself; it reflects the prespecified priority given to that comparison. The three intervals together have coverage of at least 95%, not necessarily exactly 95%.

The same inversion works for one-sided questions. For a lower confidence bound, test $H_0:\theta_i\leq\theta_i^0$ against $H_a:\theta_i>\theta_i^0$. Under the normal model, reject when $(\hat{\theta}_i-\theta_i^0)/SE_i>z_{1-\alpha_i}$. The values of $\theta_i^0$ not rejected form the lower confidence set $[\hat{\theta}_i-z_{1-\alpha_i}SE_i,\infty)$.

Worked Example: Simultaneous One-Sided Lower Bounds

A hypothetical trial has two prespecified treatment effects, measured in units of clinical benefit. The estimates are 1.8 and 0.9, with standard errors 0.6 and 0.5. Assume valid standard normal tests. For familywise one-sided level 0.05, use equal allocations $\alpha_i=0.025$. The critical value is $z_{0.975}=1.96$.

The first lower bound is $1.8-1.96(0.6)=1.8-1.176=0.624$ units. The second is $0.9-1.96(0.5)=0.9-0.980=-0.080$ units. The simultaneous confidence sets are therefore $[0.624,\infty)$ and $[-0.080,\infty)$.

With the stated validity assumptions, the probability that both bounds are below their respective true effects is at least 0.95. The second bound does not establish that its treatment effect is positive; it permits effects as low as $-0.080$ units.

Choosing and Checking an Interval Construction

The simplest comparison is between pointwise intervals and simultaneous intervals. A pointwise interval targets coverage for one effect at a time. Bonferroni intervals target coverage for the whole family and remain valid under arbitrary dependence, provided the marginal intervals are valid and the allocations sum to at most $\alpha$. A dependence-aware joint method can sometimes give narrower intervals by using the joint distribution of the estimates, but it requires that distribution to be correctly specified and implemented.

Construction	Coverage target	Main consideration
Separate pointwise intervals	Each interval individually has its stated coverage	Does not generally guarantee familywise coverage
Bonferroni intervals	All intervals cover together with probability at least $1-\alpha$	Simple and robust to dependence, but may be conservative
Inversion of a general FWER procedure	Joint parameter vector lies in the inverted set with probability at least $1-\alpha$	May require computing and projecting a non-rectangular set
Dependence-aware simultaneous intervals	All intervals cover together at the specified level	Can be more efficient, but relies on a suitable joint model

Before interpreting the intervals, check that the family of treatment effects matches the prespecified scientific question; the estimates, standard errors, and interval tails match the analysis; and the error allocations sum to no more than $\alpha$. For normal-based intervals, also check the normal approximation and the standard-error calculation. Dependence does not invalidate the Bonferroni guarantee, but using an invalid marginal test or interval does.

Adjusted p-values and intervals express related multiplicity control, but they answer different reporting questions: a p-value summarizes evidence against a null value, while an interval shows a range of effect values not excluded by a specified procedure. In general, an interval is obtained by inverting the tests across candidate null values; it is not obtained by simply attaching an unadjusted interval to an adjusted p-value after the fact.

Common Pitfalls and Limitations

Calling separate 95% intervals simultaneous 95% intervals. Each interval may have 95% marginal coverage while the probability that all cover is lower. Use a procedure designed for familywise coverage.
Interpreting the confidence level as a probability about fixed effects. The 95% statement describes the long-run coverage of the method under its assumptions, not a 95% posterior probability after observing the data.
Changing the family or allocations after seeing results. That changes the error-control claim. Define the comparisons and any unequal allocation in advance.
Assuming every inverted set is a convenient interval. General procedures can produce complicated joint regions, whose coordinate projections may be wide. Bonferroni gives a direct rectangular construction at the cost of possible conservatism.

No multiplicity adjustment repairs bias, poor measurement, incorrect standard errors, or a misspecified model. Nor does simultaneous coverage make every effect estimate precise. The benefit is a coherent statement about uncertainty across the selected family; the cost is often wider intervals than those for a single comparison.

Check Your Understanding

Try each question, then compare your reasoning with the solutions.

For four two-sided treatment-effect intervals and familywise $\alpha=0.05$, what equal component level does Bonferroni assign, and what coverage does it guarantee?
Three valid marginal intervals use allocations $0.02$, $0.02$, and $0.02$. Does the union-bound argument guarantee 95% simultaneous coverage?
A study has two one-sided effects with estimates 2.5 and standard errors 0.4 for both. Use equal Bonferroni allocation at familywise $\alpha=0.05$ to calculate each lower bound.
Why does the fact that each of three intervals has 95% pointwise coverage not establish 95% simultaneous coverage? What would the joint coverage be if the three coverage events were independent?

Solutions. (1) Each component level is $0.05/4=0.0125$. Since the allocations sum to 0.05, the probability all four intervals cover is at least 0.95. (2) No. The allocations sum to $0.06$, so the union bound guarantees only at least $1-0.06=0.94$ simultaneous coverage. (3) Each one-sided allocation is $0.025$, with critical value $z_{0.975}=1.96$. Each lower bound is $2.5-1.96(0.4)=2.5-0.784=1.716$ units. (4) Pointwise coverage applies separately to each interval and does not control the probability of any miss across all three. Under independence, joint coverage would be $0.95^3=0.8574$, or about 85.7%; without independence, the joint coverage cannot be obtained from the three marginal coverages alone.