Worked Comparison of Bonferroni, Scheffé, and Max-t

Why Compare Simultaneous Procedures in One Model?

Bonferroni, Scheffé, and max-t intervals all aim to cover multiple contrast parameters at once, but they use different information. Bonferroni controls error without using dependence among the estimates. Scheffé controls error for every linear contrast in a specified parameter space. Max-t uses the joint distribution of the particular contrasts under study. A worked comparison makes clear why their interval widths need not agree.

Throughout, consider the normal linear model \(Y=X\beta+\varepsilon\), where \(X\) is a fixed design matrix of rank \(p\), \(\varepsilon\sim N(0,\sigma^2 I_n)\), and \(\nu=n-p\). Let \(\hat{\beta}\) be the least-squares estimator and let \(s^2=\operatorname{SSE}/\nu\). For a contrast vector \(c\), the parameter is \(c'\beta\), its estimate is \(c'\hat{\beta}\), and its estimated standard error is \(s\sqrt{c'(X'X)^{-1}c}\).

Definition: For a specified family of \(m\) contrast vectors \(c_1,\ldots,c_m\), a \(1-\alpha\) simultaneous confidence procedure has simultaneous coverage at least \(1-\alpha\) if the probability that all \(m\) intervals contain their respective parameters is at least \(1-\alpha\). The family is simultaneous because the coverage statement concerns the whole collection, not each interval separately.

Three Critical Values for the Same Contrast Family

Write \(SE_j=s\sqrt{c_j'(X'X)^{-1}c_j}\) and \(\hat{\theta}_j=c_j'\hat{\beta}\). Bonferroni assigns error probability \(\alpha/m\) to each two-sided interval, giving critical value \(t_{\nu,\,1-\alpha/(2m)}\), the indicated quantile of a central \(t\) distribution with \(\nu\) degrees of freedom.

Scheffé instead protects every linear contrast in the \(p\)-dimensional coefficient space. Its critical multiplier is \(\sqrt{pF_{p,\nu;\,1-\alpha}}\), where \(F_{p,\nu;\,1-\alpha}\) is the \(1-\alpha\) quantile of the central \(F\) distribution with degrees of freedom \(p,\nu\). The family of interest may contain only a few contrasts, but Scheffé does not use that restriction.

For max-t, let \(R\) be the correlation matrix of the standardized contrast estimates. Under the normal model, the vector of statistics \(T_j=(\hat{\theta}_j-c_j'\beta)/SE_j\) has a multivariate \(t\) distribution: it can be represented as \(Z/\sqrt{U/\nu}\), where \(Z\sim N(0,R)\), \(U\sim\chi^2_\nu\), independently. The max-t critical value \(q_{1-\alpha}(R,\nu)\) is the \(1-\alpha\) quantile of \(\max_j|T_j|\). It depends on both the family’s correlation structure and the residual degrees of freedom.

Theorem: Each of the following procedures has simultaneous coverage at least \(1-\alpha\) for its stated target: Bonferroni intervals for the specified \(m\) contrasts; Scheffé intervals for all linear contrasts in the \(p\)-dimensional model space; and max-t intervals for the specified family, using the exact joint \(t\) critical value.

For Bonferroni, the coverage claim follows directly from the union bound. Each usual two-sided \(t\) interval has noncoverage probability \(\alpha/m\), so the probability that at least one interval fails is at most the sum of those probabilities, \(\alpha\). Dependence among the contrasts does not invalidate this bound.

P(\text{at least one interval fails}) =P\left(\bigcup_{j=1}^m\{\text{interval }j\text{ fails}\}\right) \leq \sum_{j=1}^m P(\text{interval }j\text{ fails}) \leq m\frac{\alpha}{m}=\alpha.

For max-t, the defining quantile gives \(P(\max_j|T_j|\leq q_{1-\alpha})=1-\alpha\) when the distribution is continuous. The event \(\max_j|T_j|\leq q_{1-\alpha}\) is exactly the event that every interval with multiplier \(q_{1-\alpha}\) covers its contrast. Thus the max-t procedure has simultaneous coverage \(1-\alpha\). The Scheffé guarantee follows from the usual normal-theory \(F\) bound for the full contrast space; the geometric derivation for this model is included below.

Worked Example: Correcting a Bonferroni Half-Width

Worked Example: A Half-Width and Two Intervals

Suppose a Bonferroni calculation for a particular contrast family gives the critical value \(2.42312\), and a contrast has estimated standard error \(\sqrt{3}\). Calculate the half-width and intervals for estimated contrasts \(2.4\) and \(-0.6\).

The half-width is the critical value multiplied by the standard error:

2.42312\sqrt{3} \approx 2.42312(1.73205) =4.196965\ldots \approx 4.197.

Thus the interval centered at \(2.4\) is \(2.4\pm4.197=(-1.797,6.597)\), to three decimal places. The interval centered at \(-0.6\) is \(-0.6\pm4.197=(-4.797,3.597)\). Each has width \(2(4.197)=8.394\), subject only to rounding of the displayed endpoints.

Worked Example: Three Correlated Contrasts in a Normal Model

Worked Example: Constructing the Contrast Family

Take \(n=27\), \(p=3\), \(X'X=I_3\), and \(s=1\), so \(\nu=24\). Use the three unit contrast vectors

c_1=(1,0,0)',\qquad c_2=\left(\frac12,\frac{\sqrt3}{2},0\right)',\qquad c_3=\left(\frac12,\frac{\sqrt3}{6},\sqrt{\frac23}\right)'.

Each vector has squared length 1. For example, \(c_3'c_3=1/4+1/12+2/3=1\). Their pairwise inner products are all \(1/2\): \(c_1'c_2=c_1'c_3=1/2\), and \(c_2'c_3=1/4+1/4=1/2\). Since \(X'X=I_3\), the estimated contrast variances are \(s^2c_j'c_j=1\), and the correlation matrix is

R= \begin{pmatrix} 1&1/2&1/2\\ 1/2&1&1/2\\ 1/2&1/2&1 \end{pmatrix}.

Suppose the observed contrast estimates are \((\hat{\theta}_1,\hat{\theta}_2,\hat{\theta}_3)=(2,-1,1.5)\). These values are compatible with a coefficient estimate: the three contrast vectors are linearly independent, so their three inner products determine a unique \(\hat{\beta}\). Every standard error is 1. The confidence intervals therefore have the form “estimate \(\pm\) critical value.”

Worked Example: Calculating and Comparing the Three Procedures

Use familywise confidence level \(0.95\), so \(\alpha=0.05\), with \(m=p=3\) and \(\nu=24\). The Bonferroni multiplier is \(b=t_{24,\,1-0.05/6}\), because each of the three two-sided intervals receives tail probability \(0.05/6\). The resulting intervals are

(2-b,2+b),\qquad(-1-b,-1+b),\qquad(1.5-b,1.5+b).

Each interval has width \(2b\). The notation for the quantile specifies its value exactly; a numerical implementation can evaluate it to the desired precision.

For Scheffé, the multiplier is \(h=\sqrt{3F_{3,24;\,0.95}}\). The intervals are

(2-h,2+h),\qquad(-1-h,-1+h),\qquad(1.5-h,1.5+h),

with common width \(2h\). For max-t, define \(q=q_{0.95}(R,24)\) as the unique solution of

\int_0^\infty P\left( |Z_1|\leq q\sqrt{u/24},\ |Z_2|\leq q\sqrt{u/24},\ |Z_3|\leq q\sqrt{u/24} \right) f_{\chi^2_{24}}(u)\,du=0.95, \qquad Z\sim N(0,R).

This equation integrates the conditional Gaussian probability over the independent chi-square denominator in the multivariate \(t\) representation. It specifies the exact critical value; it can be evaluated by numerical integration. The max-t intervals are \((2-q,2+q)\), \((-1-q,-1+q)\), and \((1.5-q,1.5+q)\), each of width \(2q\).

The max-t multiplier is no larger than the Bonferroni multiplier: at the Bonferroni threshold \(b\), the union bound gives \(P(\max_j|T_j|>b)\leq0.05\), so the 0.95 quantile of the maximum cannot exceed \(b\). Thus max-t intervals are no wider than the Bonferroni intervals in this example. Scheffé answers a broader question, protecting every contrast, rather than only these three; its width is \(2\sqrt{3F_{3,24;\,0.95}}\). Comparing the procedures means comparing their critical multipliers, not assuming that one method is always shortest for every possible family.

Why the Scheffé Bound Covers Every Contrast

The Scheffé factor is not a family-by-family adjustment. Its guarantee comes from bounding the largest standardized estimation error across the entire contrast space. Here is the normal-theory argument, with \(X\) of rank \(p\), \(\nu=n-p\), and \(\sigma^2\) estimated independently by \(s^2\).

Derivation: With probability at least \(1-\alpha\), every contrast \(c'\beta\) satisfies \(\left|c'(\hat{\beta}-\beta)\right|\leq \sqrt{pF_{p,\nu;\,1-\alpha}}\;s\sqrt{c'(X'X)^{-1}c}\).

Let \(d=\hat{\beta}-\beta\), and set \(A=X'X\), a positive-definite matrix because \(X\) has full column rank. By Cauchy–Schwarz after transforming by \(A^{1/2}\),

|c'd| =\left|(A^{-1/2}c)'(A^{1/2}d)\right| \leq\sqrt{c'A^{-1}c}\sqrt{d'Ad}.

The normal linear model gives \(d\sim N(0,\sigma^2 A^{-1})\), so \(d'Ad/\sigma^2\sim\chi^2_p\). Also \(\nu s^2/\sigma^2\sim\chi^2_\nu\), independently of \(d\). Consequently,

\frac{d'Ad}{p s^2} = \frac{(d'Ad/\sigma^2)/p}{(\nu s^2/\sigma^2)/\nu} \sim F_{p,\nu}.

On the event \(d'Ad/(p s^2)\leq F_{p,\nu;\,1-\alpha}\), which has probability \(1-\alpha\), the Cauchy–Schwarz bound holds for every \(c\) at once. Rearranging gives the stated Scheffé bound, so the simultaneous coverage extends over all linear contrasts in the model space. \(\square\)

Worked Example: What Changes When the Contrast Family Changes?

Worked Example: A Family-Specific Adjustment Versus an All-Contrasts Guarantee

Keep the same model, but suppose the scientific question concerns only \(c_1'\beta\) and \(c_2'\beta\), not \(c_3'\beta\). For Bonferroni, \(m\) changes from 3 to 2. The multiplier becomes \(t_{24,\,1-0.05/4}\), and the two intervals have width \(2t_{24,\,1-0.05/4}\). The pair remains correlated, with correlation \(1/2\), so a max-t calculation for this pair uses the corresponding \(2\)-by-\(2\) correlation matrix and its own maximum-statistic quantile.

Scheffé does not change: it still protects every contrast in the three-dimensional coefficient space and uses \(\sqrt{3F_{3,24;\,0.95}}\). This is appropriate if contrasts might be selected after examining the data or if a simultaneous guarantee for any direction is genuinely required. If the two contrasts were fixed in advance and are the only inferential targets, a family-specific method can avoid paying for protection over a much larger set.

Common Mistake: Treating Dependence as Irrelevant to Every Method

A frequent error is to say, “The contrasts are correlated, so Bonferroni is invalid,” or, in the opposite direction, “Bonferroni accounts for the correlation because it controls familywise error.” Both statements are wrong. Bonferroni remains valid under arbitrary dependence, but it does not use the dependence to sharpen its critical value. Max-t uses the joint distribution, so its critical value can reflect the correlation structure. Scheffé protects a different target: the whole contrast space, not merely the listed contrasts.

Another important distinction is between marginal and simultaneous coverage. Three separate 95% intervals do not generally provide 95% probability that all three cover. The procedures compared here calibrate the collection to achieve the stated familywise coverage. Their widths differ because they pay for that guarantee in different ways.

Key takeaway: Use Bonferroni for a simple dependence-robust guarantee over a specified family, max-t when the exact joint distribution for that family is available, and Scheffé when protection over all linear contrasts is the inferential goal. Compare widths only after specifying the family and the coverage target.

Check Your Understanding

Answer each question by identifying the family being protected and the critical value that corresponds to it.

For \(m=4\) two-sided contrasts and familywise error level \(\alpha\), what \(t\)-quantile does the Bonferroni procedure use?
Why does correlation among contrast estimates not invalidate Bonferroni intervals?
What additional distributional information does max-t use that Bonferroni does not?
In a model with \(p=5\) coefficients and \(\nu=30\) residual degrees of freedom, write the Scheffé multiplier for confidence level \(1-\alpha\).
If the scientific target changes from three prespecified contrasts to every linear contrast, which procedure’s target already matches that broader question?