Scheffé Adjustments for Flexible Contrasts

Why Flexible Contrasts Need Simultaneous Protection

After an ANOVA, a useful question may compare two groups. But a scientific question can also compare one group with the average of several others, or compare two different averages. If the comparisons are chosen only after looking at the sample means, ordinary single-comparison intervals do not automatically protect the overall family of conclusions. Scheffé’s method addresses this by providing simultaneous inference for every contrast among the group means.

That flexibility is valuable when the relevant contrasts were not all specified in advance. It also comes at a cost: Scheffé intervals can be wider than intervals from procedures designed for a narrower family, such as pairwise comparisons. The method is not a correction to the group means themselves; it changes the uncertainty threshold used for inference.

Model, Contrasts, and Their Standard Errors

Suppose a one-way ANOVA compares \(k\) groups. Let \(\mu_i\) be the population mean response in group \(i\), \(n_i\) its sample size, and \(N=\sum_{i=1}^k n_i\) the total sample size. The model assumes independent observations, normally distributed responses within each group, and a common population variance \(\sigma^2\) across groups. The pooled within-group variance estimate is the ANOVA mean square error, denoted \(MSE\), with \(df=N-k\) error degrees of freedom.

These assumptions are needed for the standard Scheffé results. They should be plausible given the study design and the data; Scheffé’s adjustment does not repair dependence, serious model problems, or unequal group variances.

The coefficients must sum to zero so that a contrast describes a comparison among means rather than their overall level. For example, \(\mu_1-\mu_2\) has coefficients \(1,-1,0,\ldots,0\). The comparison of the average of groups 1 and 2 with group 3 is \((\mu_1+\mu_2)/2-\mu_3\), with coefficients \(1/2,1/2,-1\).

Scheffé’s procedure uses the same estimate and standard error as ordinary contrast inference. Its distinctive feature is the multiplier applied to that standard error. For a familywise error rate \(\alpha\), define \(F_{1-\alpha;k-1,N-k}\) as the \(1-\alpha\) quantile of an \(F\) distribution with \(k-1\) and \(N-k\) degrees of freedom.

“Simultaneous” is the key word. The guarantee applies to the whole set of contrasts, not merely to one contrast selected before data collection. It therefore remains valid if a researcher first inspects the results and then chooses a contrast to report. The guarantee is about long-run coverage under the stated model, not a probability assigned to a fixed parameter after the interval has been calculated.

Worked Example: A Difference Between Two of Four Means

Worked Example: Comparing an Average With Another Group

How the Simultaneous Guarantee Works

The critical multiplier comes from the geometry of the \(k\) group means. The following justification explains why it covers contrasts even when their coefficients are selected flexibly. Let \(d_i=\bar{x}_i-\mu_i\) be the estimation error for group \(i\), and let \(\bar d_w=\sum_i n_i d_i/N\) be its sample-size-weighted average. For any contrast, \(\sum_i c_i=0\), so

Applying the Cauchy–Schwarz inequality to the last sum gives

The right-hand side measures estimation error in the \(k-1\) dimensional space of contrasts. Under the model, its ratio to \((k-1)MSE\) has an \(F\) distribution with \(k-1\) and \(N-k\) degrees of freedom. Thus, with probability at least \(1-\alpha\),

Combining the inequalities and taking square roots shows that, at once for every contrast,

Rearranging this bound gives the Scheffé interval. Since the same bound applies to every coefficient vector summing to zero, it also covers contrasts chosen after examining the data.

Testing a Contrast and Comparing Methods

To test \(H_0:L=L_0\), construct the Scheffé interval for \(L\). Reject \(H_0\) at familywise level \(\alpha\) if the interval does not contain \(L_0\). Equivalently, reject when

For a test against zero, the interval excludes zero exactly when this inequality holds with \(L_0=0\). The corresponding decision is protected as part of the full family of contrasts, rather than as an isolated test.

Different multiple-comparison procedures protect different families. Tukey procedures focus on all pairwise differences, while Bonferroni and Holm adjustments protect a finite set of selected comparisons. Scheffé’s method protects the much broader family of all contrasts. That breadth is useful when a contrast may be formulated after seeing the results, but it generally makes Scheffé intervals more conservative for a pairwise-only question. The method should be chosen to match the inferential family, not simply because it is available.

Common Mistake: Treating Scheffé as a Pairwise Adjustment

A common error is to use a pairwise critical value, or an ordinary single-contrast \(t\) critical value, and then claim protection for any contrast selected from the data. Those thresholds do not provide Scheffé’s guarantee for the full family. The correction is to use \(\sqrt{(k-1)F_{1-\alpha;k-1,N-k}}\) with the contrast’s standard error, using \(k-1\) numerator and \(N-k\) denominator degrees of freedom. Also check that the coefficients sum to zero and use \(MSE\) from the same one-way ANOVA model.