Power of Bonferroni, Scheffé, and Max-t Procedures

Power Depends on Both the Alternative and the Protection Goal

Simultaneous procedures are designed to control the probability of making one or more false rejections across a family of contrasts. That protection affects power: a larger critical value makes rejection harder. But comparing procedures requires more than comparing critical values. One must also specify the family of contrasts, their dependence, and what counts as success under the alternative.

For a specified finite family, a max-t procedure uses the joint distribution of its contrast statistics, whereas Bonferroni uses a bound that does not require their dependence structure. Scheffé protection is broader: it covers every contrast in a contrast space, not just a prespecified finite list. That wider guarantee can be costly when the scientific target is a small finite family.

Use the usual exact normal-theory setting: the contrast estimates are jointly normal, and the common error variance is estimated independently by \(s^2\), with \(\nu\) error degrees of freedom. Let \(q\) be the dimension of the contrast space, and write \(v_j\) for the variance factor of contrast \(C_j\), so that \(\operatorname{Var}(\hat C_j)=\sigma^2v_j\). Under an alternative, define the standardized effect \(\delta_j=(C_j-\theta_{j,0})/(\sigma\sqrt{v_j})\), where \(\theta_{j,0}\) is the null value. Let \(R\) be the correlation matrix of the standardized contrast errors.

This representation gives an exact route from an alternative to family power. For a common two-sided cutoff \(c\), conditional on \(W=w\), rejection occurs if at least one coordinate satisfies \(|Z_j+\delta_j|>c\sqrt{w/\nu}\). Therefore

The formula retains the dependence among contrasts and the shared variance estimate. Treating the studentized statistics as independent t variables generally gives the wrong family power.

Critical Values and What They Protect

For two-sided tests at familywise level \(\alpha\), the Bonferroni cutoff is \(t_{\nu,1-\alpha/(2m)}\). The max-t cutoff is the \(1-\alpha\) quantile of \(\max_j |Z_j|/\sqrt{W/\nu}\) under the joint null, using the family’s correlation matrix. Scheffé’s cutoff for the full \(q\)-dimensional contrast space is \(\sqrt{qF_{q,\nu;1-\alpha}}\), where \(F_{q,\nu;1-\alpha}\) is the \(1-\alpha\) quantile of the central \(F_{q,\nu}\) distribution.

These choices have different guarantees. Bonferroni controls the familywise error rate for the specified \(m\) tests, regardless of their dependence. Max-t uses the joint distribution for that finite family and controls its familywise error rate. Scheffé controls the probability of any false rejection over the entire \(q\)-dimensional contrast space. Consequently, a finite-family max-t cutoff can be smaller than the Scheffé cutoff: the maximum over a finite set cannot exceed the maximum over all directions.

For the first comparison, under the null each statistic has a t distribution with \(\nu\) degrees of freedom. The union bound gives \(P(\max_j |T_j|>c_B)\leq\sum_jP(|T_j|>c_B)=m(\alpha/m)=\alpha\), where \(c_B=t_{\nu,1-\alpha/(2m)}\). The max-t cutoff is the smallest cutoff achieving the specified null probability \(\alpha\), so it cannot be larger than \(c_B\).

For the second comparison, consider a contrast-space estimate represented in \(q\) orthonormal coordinates. The squared maximum absolute studentized contrast over all unit directions is \(qF\), with \(F\sim F_{q,\nu}\) under the null. Restricting the maximum to a finite set of directions cannot increase it. Hence the finite-family maximum has a critical value no larger than the full-space Scheffé critical value. These are ordering results for critical values, not a universal ordering between Bonferroni and Scheffé; their relative sizes depend on \(m\), \(q\), \(\nu\), and \(\alpha\).

Worked Power Comparisons

How to Compare Power Correctly

A fair comparison holds the alternative, significance level, and target family fixed, then computes the relevant rejection probability under each rule. For the any-rejection criterion, the target is \(P(\bigcup_j R_j)\). For all-rejections power, the target is \(P(\bigcap_j R_j)\), which can be much smaller, especially when some contrasts have effects near zero. A procedure can have respectable any-rejection power while rarely rejecting every member of a family.

The contrast correlation matters because max-t calibrates to the distribution of the maximum. Strong positive dependence can make simultaneous extreme values less likely than under independence, often lowering the max-t critical value. Negative or more complex dependence can change the joint tail probability differently. Bonferroni does not adapt to this dependence, while Scheffé pays for protecting all directions rather than just the chosen contrasts.

Common Pitfall

A common mistake is to say, “Scheffé is more conservative, so it always has lower power than Bonferroni.” This does not follow. Scheffé’s cutoff may be larger or smaller than the Bonferroni cutoff, depending on the number of tests and the contrast-space dimension. Moreover, power depends on the alternative’s direction and on the criterion being measured. The reliable comparison is to calculate the cutoffs for the same setting and evaluate the same joint rejection event under the same alternative.

A second error is to infer family power from one contrast’s marginal power. In the first example, the Scheffé marginal power for the shifted contrast is about \(0.710\), but any-rejection power is about \(0.714\) because the null second contrast can also reject. For all-rejections power, the same null contrast instead makes success unlikely. The event in the probability statement must match the scientific question.