Comparing Bonferroni and Dunnett Sequential Boundaries

Why compare these boundaries?

In a multi-arm, multi-stage trial, a treatment may be declared effective if its statistic crosses a boundary at any planned analysis. Testing several treatments at several stages creates multiple opportunities for a false rejection. Bonferroni and Dunnett-type boundaries both address this multiplicity, but they use different information: Bonferroni protects against every test without using dependence, whereas Dunnett-type calibration accounts for the correlation created by a shared control and repeated analyses.

The comparison below holds the design and familywise significance level fixed. This makes the role of the joint distribution visible. The numerical results are hypothetical and depend on the stated model; they are not universal constants for three-arm, two-stage trials.

Design and joint model

Consider three experimental treatments, indexed by \(i=1,2,3\), compared with one shared control at two planned analyses, indexed by \(j=1,2\). Let \(Z_{ij}\) be the standardized, upper-tail statistic for treatment \(i\) versus control at analysis \(j\). The efficacy hypotheses are one-sided: for each treatment, the null is that its effect is no greater than zero, and the alternative is that it is greater than zero. A rejection occurs if the statistic crosses its efficacy boundary at either analysis.

Model and assumptions: Under the joint normal model, the six statistics have unit marginal variances and can be written as \(Z_{ij}=\mu_{ij}+\varepsilon_{ij}\), where the error vector \(\varepsilon\) is multivariate normal with mean zero and a prespecified covariance matrix. Under the complete null, all \(\mu_{ij}=0\). For a true one-sided null, assume its statistic means satisfy \(\mu_{ij}\leq 0\). The covariance structure is that induced by a shared control and the planned cumulative analyses; the example assumes equal treatment-control correlation and equal information fractions across arms. The test statistics are assumed to be standardized as specified, with no unaccounted variance estimation or model misspecification.

For this example, the information fraction at the first analysis is \(t_1=0.5\), and at the final analysis it is \(t_2=1\). The canonical repeated-analysis correlation is \(\sqrt{t_1/t_2}=\sqrt{0.5}\). With equal allocation and a shared control, statistics for different treatments at the same analysis have correlation \(0.5\). More generally, the covariance between statistics for different treatments is the treatment-control correlation multiplied by the canonical stage correlation.

\operatorname{Corr}(Z_{ij},Z_{i'j'}) = \begin{cases} \sqrt{\min(t_j,t_{j'})/\max(t_j,t_{j'})}, & i=i',\\ 0.5\sqrt{\min(t_j,t_{j'})/\max(t_j,t_{j'})}, & i\ne i'. \end{cases}

Thus the within-treatment correlation across the two analyses is about \(0.7071\); the cross-treatment correlation is \(0.5\) at a common analysis and about \(0.3536\) across different analyses. These values are design assumptions, not a universal property of MAMS trials.

Two ways to set the boundary

Bonferroni sequential boundary: If the total one-sided familywise error target is \(\alpha\), and there are \(K\) treatments and \(J\) planned analyses, a simple Bonferroni rule uses the same marginal upper-tail probability \(\alpha/(KJ)\) at each treatment-analysis pair. Its critical value is \(c_B=\Phi^{-1}(1-\alpha/(KJ))\), where \(\Phi\) is the standard normal distribution function.

This rule guarantees that the probability of at least one false rejection is at most \(\alpha\), regardless of dependence among the tests, provided each true-null marginal upper-tail probability is no larger than \(\alpha/(KJ)\). It can be conservative because the probabilities of the individual rejection events are added even when those events overlap substantially.

Dunnett-type sequential boundary: Let \(M=\max_{i,j} Z_{ij}\). A common-boundary Dunnett-type rule chooses \(c_D\) so that, under the complete-null joint distribution, \(P(M\geq c_D)=\alpha\). A treatment is rejected if its statistic crosses \(c_D\) at either planned analysis.

This is a joint calibration of the maximum, not a separate single-stage Dunnett adjustment repeated without accounting for the interim looks. The correlation across both arms and stages must be included. Numerical integration or a multivariate-normal probability routine can solve the calibration equation.

Why the two rules control error

The Bonferroni guarantee follows from the union bound. Let \(A_{ij}\) be the event that statistic \(Z_{ij}\) crosses \(c_B\) when treatment \(i\) is a true null. If there are \(m\leq KJ\) such tests, then

P\left(\bigcup_{(i,j)\text{ true null}} A_{ij}\right) \leq \sum_{(i,j)\text{ true null}}P(A_{ij}) \leq m\frac{\alpha}{KJ} \leq \alpha.

The Dunnett-type calibration requires a more careful argument for strong control, meaning control under every configuration of true and false null hypotheses. Calibration directly gives error \(\alpha\) under the complete null, but under a partial-null configuration the true-null statistics need not have zero means. The appropriate argument uses the specified covariance and the nonpositive means of true-null statistics.

Form the true-null subset.
Let \(I_0\) be the set of true-null treatments. A false rejection is possible only if some \(Z_{ij}\), with \(i\in I_0\), crosses the boundary.

Use a common-error coupling.
Under the model, represent the true-null statistics as \(Z_{ij}=\mu_{ij}+\varepsilon_{ij}\), using the specified covariance for their errors, with \(\mu_{ij}\leq 0\). For each realization of the errors, \(Z_{ij}\leq \varepsilon_{ij}\) coordinatewise.

Bound by the complete-null maximum.
If any true-null statistic crosses \(c_D\), then the corresponding zero-mean error statistic also crosses it. The maximum over the true-null subset cannot exceed the maximum over all \(KJ\) zero-mean statistics under the same covariance model.

Apply the calibration.
The probability of a false rejection is therefore no greater than the complete-null probability \(P(\max_{i,j}\varepsilon_{ij}\geq c_D)=\alpha\).

This argument does not claim that the true-null statistics have the centered complete-null distribution when their means are negative. Instead, it uses coordinatewise monotonicity of an upper-tail rejection event and the subset relationship between maxima. Strong control depends on the stated joint model, including the covariance structure and nonpositive true-null means.

Worked comparison for the same design

Worked Example: Calibrating Both Boundaries

Suppose the trial has \(K=3\) treatments, \(J=2\) analyses, and one-sided familywise target \(\alpha=0.025\). Use the information fractions and correlations specified above.

Bonferroni calculation. There are \(KJ=3(2)=6\) treatment-analysis tests, so the marginal tail area is

\frac{\alpha}{KJ}=\frac{0.025}{6}=0.004167.

The standard normal critical value is

c_B=\Phi^{-1}(1-0.004167)=\Phi^{-1}(0.995833)\approx 2.638.

Each treatment must therefore have \(Z_{ij}\geq 2.638\) at one of the two analyses to be rejected under this rule.

Dunnett-type calculation. Use the six-dimensional covariance matrix with diagonal entries 1, within-treatment across-stage correlations 0.7071, same-stage cross-treatment correlations 0.5, and different-treatment, different-stage correlations 0.3536. Solve

P_0\left(\max_{\substack{i=1,2,3\\j=1,2}} Z_{ij}\geq c_D\right)=0.025,

where \(P_0\) denotes probability under the centered complete-null model. Numerical multivariate-normal integration for this specified matrix gives a common critical value of approximately \(c_D=2.52\) (rounded to two decimal places). The smaller value than \(2.638\) reflects the positive dependence: exceedances at different tests overlap, so six marginal tail probabilities of \(0.004167\) are more conservative than a joint calibration of the maximum.

This computed value is tied to this covariance matrix and the common-boundary rule. Changing the allocation, information fractions, number of stages, or correlation structure requires recalibration; it is not valid simply to reuse \(2.52\).

Worked Example: Applying the Calibrated Boundaries

Suppose hypothetical observed statistics are as follows. The values are standardized test statistics, not treatment effects; no effect-size units are attached to them.

Treatment	Analysis 1	Analysis 2
1	2.60	2.10
2	1.40	2.30
3	0.80	1.70

The largest statistic is \(2.60\), for treatment 1 at analysis 1. Under Bonferroni, \(2.60<2.638\), so no treatment crosses the boundary. Under the Dunnett-type rule, \(2.60>2.52\), so treatment 1 crosses the common boundary and is rejected. The difference is a consequence of the different calibrations, not a change in the observed data.

A boundary crossing is a decision under the specified testing procedure. It does not by itself quantify the treatment effect, establish clinical importance, or guarantee that the effect is positive in the underlying population.

Worked Example: Checking Strong Control in a Partial-Null Configuration

Suppose treatments 1 and 3 are true nulls, while treatment 2 has a beneficial effect. Under the model, let the true-null means at the two analyses be \((\mu_{11},\mu_{12})=(-0.20,-0.10)\) and \((\mu_{31},\mu_{32})=(0,-0.15)\). These are illustrative means on the standardized statistic scale.

For every realization of the error vector, \(Z_{11}=-0.20+\varepsilon_{11}\leq\varepsilon_{11}\), \(Z_{12}=-0.10+\varepsilon_{12}\leq\varepsilon_{12}\), \(Z_{31}=\varepsilon_{31}\), and \(Z_{32}=-0.15+\varepsilon_{32}\leq\varepsilon_{32}\). Therefore, if either true-null treatment crosses \(c_D\), the corresponding zero-mean statistic crosses it as well. The maximum of these four zero-mean statistics is no greater than the maximum of all six zero-mean statistics, including treatment 2's coordinates under the calibration model. Hence

P(\text{reject at least one true null}) \leq P_0\left(\max_{i=1,2,3;\,j=1,2}Z_{ij}\geq c_D\right) =0.025.

The argument uses the true-null means being nonpositive and the specified covariance for their statistics. It does not treat the true-null subset as if it necessarily had the full centered six-statistic distribution.

Diagnostics, alternatives, and limitations

Before using either boundary, verify the analysis plan and covariance assumptions. In particular, check that the same treatment-control contrasts and variance standardizations are used at each look, that the information fractions match the planned analyses, and that shared-control allocation is reflected in the cross-treatment correlations. A covariance matrix that is not positive semidefinite, or a matrix built with correlations inconsistent with the allocation and information structure, signals a calibration error. Simulation under the complete null can check whether the implemented Dunnett-type rule has rejection probability near \(\alpha\); simulation under several partial-null configurations can probe the strong-control argument and implementation.

Bonferroni is the main robust alternative when the joint covariance is uncertain or when simplicity is more important than power. Its union-bound guarantee does not require independent tests, but it can be conservative. Dunnett-type calibration typically gains power when treatment comparisons share control and the resulting positive correlations are correctly specified. It is not automatically safer: under a wrong covariance model, its nominal complete-null calibration may be wrong, and its strong-control justification may not apply.

Do not calibrate each analysis in isolation. A single-stage Dunnett critical value does not automatically control error across repeated looks. The joint calibration must include across-stage correlations.
Do not claim centered true-null statistics under partial nulls. A true one-sided null may have a negative effect and hence a negative statistic mean. Use the nonpositive-mean monotonicity argument, not an equality of distributions.
Do not infer that correlation always has the same numerical effect. The maximum distribution depends on the full covariance matrix. Recompute the boundary when the design changes.
Do not confuse familywise control with marginal testing. The target \(\alpha=0.025\) concerns the probability of one or more false rejections in the family, not the error probability of each test separately.

Key takeaway: Bonferroni divides the error budget across all planned treatment-stage tests and controls familywise error by the union bound. A Dunnett-type sequential boundary calibrates the joint maximum and can be less conservative when the shared-control covariance is correctly specified. Strong control requires attention to the true-null means as well as the covariance.

Check Your Understanding

Answer each question, then compare your reasoning with the brief solutions.

For four treatments at two analyses with \(\alpha=0.02\), what marginal tail probability does the equal-allocation Bonferroni rule use?
Why does a Dunnett-type calibration for two analyses need across-stage correlations?
For a one-sided true null with a negative mean, why is its rejection probability bounded using the zero-mean model?
If positive shared-control correlation is ignored when calibrating the Dunnett-type maximum, what should be done before relying on the nominal boundary?

Solutions. (1) There are \(4(2)=8\) tests, so the tail probability is \(0.02/8=0.0025\) per test. (2) The rejection event is a maximum over all treatment-stage statistics, whose joint distribution depends on their correlations; omitting the across-stage structure changes that maximum distribution. (3) Couple the statistic as \(Z=\mu+\varepsilon\), with \(\mu\leq0\); then \(Z\leq\varepsilon\) for every realization, so an upper-tail crossing under the true null implies a crossing by the corresponding zero-mean statistic. (4) Rebuild the covariance model from the design and recalibrate, then check the implementation through null-configuration simulation; the nominal boundary should not be assumed valid under an unverified covariance matrix.