Width Tradeoffs Across Simultaneous Procedures

Why Compare Widths?

A simultaneous procedure protects a collection of contrasts together, rather than protecting each interval separately. That protection has a precision cost: the intervals are generally wider than marginal intervals with the same nominal confidence level. The size of that cost depends not only on how many contrasts are listed, but also on their dependence and the dimension of the space they occupy.

To compare procedures fairly, hold the contrast family, standard errors, and familywise error level fixed. Then the procedures differ through their critical values. A smaller critical value means narrower intervals, but only if the procedure actually provides the stated simultaneous coverage under the model.

A Common Framework for Widths

Suppose \(\hat{\mu}\) estimates a mean vector \(\mu\), and the fixed family consists of \(m\) contrasts \(c_j^T\mu\), \(j=1,\ldots,m\). For a concrete comparison, first assume \(\hat{\mu}\sim N_p(\mu,\Sigma/n)\), with known positive-definite \(\Sigma\). The standard error of contrast \(j\) is

s_j=\sqrt{\frac{c_j^T\Sigma c_j}{n}}.

Write \(Z_j=\{c_j^T\hat{\mu}-c_j^T\mu\}/s_j\). Each \(Z_j\) has a standard normal distribution, and the vector of these statistics has a correlation matrix \(R\) determined by the contrasts and \(\Sigma\).

Definition: A simultaneous interval procedure with critical value \(q\) reports, for every contrast \(j\), the interval \(c_j^T\hat{\mu}\pm q s_j\). Its half-width is \(q s_j\), and its full width is \(2q s_j\). The procedure has familywise confidence level \(1-\alpha\) if the probability that all \(m\) intervals cover their respective contrasts is at least \(1-\alpha\).

When procedures use the same \(s_j\), comparing their widths reduces to comparing their critical values. The contrast-specific standard errors still matter for the actual interval lengths: a single multiplier does not make all intervals equally wide when the contrasts have different standard errors.

Four Critical-Value Strategies

For two-sided Bonferroni intervals, use

q_{\mathrm{Bonf}}=\Phi^{-1}\!\left(1-\frac{\alpha}{2m}\right),

where \(\Phi\) is the standard normal distribution function. The union bound guarantees familywise error at most \(\alpha\), regardless of dependence. Bonferroni uses the listed family size \(m\), but does not use the correlation structure.

If the \(m\) standardized statistics are mutually independent, the Šidák critical value is

q_{\mathrm{Sidak}}=\Phi^{-1}\!\left(\frac{1+(1-\alpha)^{1/m}}{2}\right).

Indeed, for a common critical value \(q\), the probability that all independent statistics satisfy \(|Z_j|\leq q\) is \(\{2\Phi(q)-1\}^m\). Setting this equal to \(1-\alpha\) and solving for \(q\) gives the expression above. Independence is essential to this exact calculation.

The exact max procedure uses the distribution of the largest absolute standardized error:

M=\max_{1\leq j\leq m}|Z_j|, \qquad q_{\max}=\text{the }(1-\alpha)\text{ quantile of }M.

For a Gaussian vector, this critical value depends on \(R\). It can be computed from the joint distribution of the contrasts; it is not determined by \(m\) alone.

Scheffé’s method uses the dimension of the contrast space. Let \(r\) be the rank of the vectors \(\Sigma^{1/2}c_1,\ldots,\Sigma^{1/2}c_m\). Its critical value for known \(\Sigma\) is

q_{\mathrm{Sch}}=\sqrt{\chi^2_{r,\,1-\alpha}},

where \(\chi^2_{r,\,1-\alpha}\) is the \(1-\alpha\) quantile of a chi-square distribution with \(r\) degrees of freedom. Scheffé protects every contrast in the span of the family, not merely the listed contrasts.

What the Critical Values Guarantee

Theorem: Under the known-covariance Gaussian model, for any fixed family, \(q_{\max}\leq q_{\mathrm{Bonf}}\) and \(q_{\max}\leq q_{\mathrm{Sch}}\). If the standardized contrasts are independent, \(q_{\max}=q_{\mathrm{Sidak}}\), and \(q_{\mathrm{Sidak}}\leq q_{\mathrm{Bonf}}\).

Proof. At the Bonferroni critical value, each event \(\{|Z_j|>q_{\mathrm{Bonf}}\}\) has probability \(\alpha/m\). Therefore, by the union bound,

P(M>q_{\mathrm{Bonf}}) \leq \sum_{j=1}^m P(|Z_j|>q_{\mathrm{Bonf}}) =m\frac{\alpha}{m}=\alpha.

Thus the \(1-\alpha\) quantile of \(M\) cannot exceed \(q_{\mathrm{Bonf}}\). If the statistics are independent, their joint probability of lying within \([-q,q]\) is \(\{2\Phi(q)-1\}^m\); setting this probability to \(1-\alpha\) proves \(q_{\max}=q_{\mathrm{Sidak}}\). Also, \((1-\alpha)^{1/m}\leq 1-\alpha/m\), since \(x^{1/m}\) is concave on \([0,1]\). Hence the Šidák per-tail probability is larger than \(\alpha/(2m)\), so its normal critical value is no larger than Bonferroni’s.

For Scheffé’s bound, set \(W=\sqrt{n}\,\Sigma^{-1/2}(\hat{\mu}-\mu)\), so \(W\sim N_p(0,I_p)\). Each standardized contrast is the inner product of \(W\) with a unit vector in the span of the whitened contrast vectors. If \(P\) is the orthogonal projection onto that span, then

M\leq \|PW\|, \qquad \|PW\|^2\sim\chi^2_r.

The first inequality follows because the absolute inner product with any unit vector in the span is at most the length of the projection onto that span. Consequently \(P(M\leq q_{\mathrm{Sch}})\geq 1-\alpha\), and \(q_{\max}\leq q_{\mathrm{Sch}}\). \(\square\)

There is no general ordering between Scheffé and Bonferroni for every family: one depends on rank and the other on the number of listed contrasts. Nor is there a universal ordering between Šidák and the exact max critical value when the statistics are dependent. Šidák’s expression is an exact max critical value only under independence.

Worked Example: Three Independent Contrasts

Let \(\alpha=0.05\), and suppose there are three independent standardized Gaussian contrasts, each with standard error \(0.40\). Their whitened contrast vectors have rank \(r=3\). Calculate the common half-width under each procedure.

For Bonferroni, the upper normal probability is \(1-0.05/(2\cdot3)=0.991667\), so \(q_{\mathrm{Bonf}}\approx2.394\). The half-width is \(2.394(0.40)=0.958\).

For Šidák, the per-contrast familywise error allocation is \(1-0.95^{1/3}\approx0.01695\). Thus \(q_{\mathrm{Sidak}}=\Phi^{-1}(1-0.01695/2)\approx2.389\), giving half-width \(2.389(0.40)\approx0.956\). Since the contrasts are independent, the exact max procedure has this same critical value.

For Scheffé, \(\chi^2_{3,0.95}\approx7.815\), so \(q_{\mathrm{Sch}}=\sqrt{7.815}\approx2.796\), and the half-width is \(2.796(0.40)\approx1.118\). Scheffé is wider here because it protects all directions in a three-dimensional contrast space, rather than just these three independent contrasts.

Worked Example: Four Listed Contrasts but Only Two Distinct Directions

Suppose four standardized contrasts have errors \((Z_1,Z_2,Z_1,Z_2)\), where \(Z_1\) and \(Z_2\) are independent standard normals. All four standard errors are \(0.50\), and \(\alpha=0.05\). There are \(m=4\) listed contrasts, but the contrast space has rank \(r=2\).

Bonferroni uses \(q=\Phi^{-1}(1-0.05/8)\approx2.498\), producing half-width \(2.498(0.50)=1.249\). Šidák applied to all four listed statistics, under its independence formula, gives \(q=\Phi^{-1}\{(1+0.95^{1/4})/2\}\approx2.49\), or a half-width of about \(1.25\). That Šidák calculation is not exact here: the duplicated statistics are not mutually independent.

The maximum over the four statistics is actually \(\max(|Z_1|,|Z_2|)\). Its exact critical value is the independent Šidák value for two, \(\Phi^{-1}\{(1+\sqrt{0.95})/2\}\approx2.234\), giving half-width \(2.234(0.50)=1.117\). Scheffé gives \(\sqrt{\chi^2_{2,0.95}}=\sqrt{5.991}\approx2.448\), or half-width \(1.224\). The duplicate contrasts increase \(m\), but neither the maximum nor the rank increases when those duplicates are added.

Worked Example: A Rank-One Family with Different Standard Errors

Let the parameter be scalar, with \(\hat{\mu}\sim N(\mu,0.04)\), and consider the three contrasts \(\mu\), \(2\mu\), and \(3\mu\). Their standard errors are \(0.20\), \(0.40\), and \(0.60\), respectively. The standardized errors are all the same random variable up to sign, so the family has rank one and its maximum absolute error is just \(|Z|\).

At \(\alpha=0.05\), the exact max critical value and the rank-one Scheffé critical value are both \(1.96\). The three exact simultaneous half-widths are \(1.96(0.20)=0.392\), \(1.96(0.40)=0.784\), and \(1.96(0.60)=1.176\).

Bonferroni still uses \(m=3\), giving \(q\approx2.394\) and half-widths \(0.479\), \(0.958\), and \(1.436\). The numerical comparison separates two effects: the common critical value reflects multiplicity and dependence, while each standard error scales its own interval.

When the Variance Is Estimated

In a normal linear model with a common estimated variance and \(\nu\) residual degrees of freedom, the standardized contrast statistics have a multivariate \(t\) distribution. Bonferroni then uses \(t_{\nu,\,1-\alpha/(2m)}\); Scheffé uses \(\sqrt{rF_{r,\nu,\,1-\alpha}}\); and an exact max-\(t\) procedure uses the quantile of the maximum absolute value of that multivariate \(t\) vector. These are the corresponding finite-sample critical values under the model assumptions.

A common mistake is to use the Šidák formula merely because the contrast estimates have zero correlation. With a shared estimated variance, the statistics share the same random denominator, so zero correlation of their Gaussian numerators does not generally make the \(t\) statistics independent. The independence condition must apply to the statistics whose maximum is being calibrated. Otherwise, use an exact max-\(t\) calculation for the joint distribution or a valid Bonferroni procedure.

How to Interpret the Tradeoff

Bonferroni is dependable when the family is fixed and a simple dependence-robust guarantee is valuable, but it can pay a substantial cost when many contrasts are redundant. Šidák can be slightly narrower than Bonferroni when its independence condition holds. Exact max-t uses the dependence structure directly and is no wider than Bonferroni under the stated Gaussian setup. Scheffé is attractive when protection for every contrast in a low-rank space is scientifically useful; for a small selected family, that broader protection may make its intervals unnecessarily wide.

The family must be fixed for these comparisons. Changing the family after inspecting the data changes the inferential problem and is not addressed by simply selecting the smallest critical value in this comparison. For a fixed family, report the family definition, the method, its assumptions, and the resulting interval widths. In particular, state whether a quoted Šidák value is justified by independence and whether a max-t value uses the actual dependence structure.

Key takeaway: Width is the product of a contrast-specific standard error and a simultaneous critical value. Bonferroni responds to listed family size, Scheffé to contrast-space rank, and exact max-t to the joint dependence; Šidák matches the exact maximum only when the relevant statistics are independent.

Check Your Understanding

For each question, distinguish the family size, contrast-space rank, and dependence assumptions.

For a fixed family of independent standard normal contrasts, why is the exact max critical value the Šidák critical value?
Why can adding an exact duplicate contrast increase the Bonferroni critical value without changing the exact maximum?
What does Scheffé protect beyond a finite list of contrasts, and which dimension determines its known-covariance critical value?
Can zero correlation of Gaussian numerators alone justify Šidák adjustment for \(t\) statistics that share an estimated variance? Explain.
For procedures using the same contrast standard errors, what does a smaller critical value imply about interval widths, and what must still be checked before preferring that procedure?