Why Small Errors Can Change an ANOVA Conclusion
A contrast or comparison is a targeted question about population means. Its arithmetic can look straightforward, but a wrong coefficient set, an incorrect standard error, or an unaccounted-for search across comparisons can change the conclusion. These errors are especially easy to make when the groups have unequal sample sizes or when the comparison of interest is chosen after viewing the results.
The setting here is the one-way fixed-effects ANOVA model. For group \(i\), observations are independent and normally distributed with population mean \(\mu_i\) and a common variance \(\sigma^2\). The usual pooled error mean square, \(MSE\), estimates that common variance, and the error degrees of freedom are \(N-k\), where \(N\) is the total sample size and \(k\) is the number of groups. Inference using these formulas relies on independent observations, normal errors within groups, and equal group variances.
The zero-sum condition makes the quantity a comparison among means rather than a comparison mixed with their overall level. A general linear combination with coefficients that do not sum to zero may still be mathematically defined, but it is not a contrast. For example, adding the same constant to every population mean leaves a contrast unchanged; it does not leave a nonzero-sum linear combination unchanged.
Worked Example: Repairing a Noncontrast
Worked Example: Repairing a Noncontrast
A hypothetical materials study compares mean fatigue lives, in thousands of cycles, for three treatment groups. The observed means are \(\bar Y_1=10\), \(\bar Y_2=14\), and \(\bar Y_3=20\). An analyst proposes coefficients \((1,-1,1)\) to compare the third group with the first two. Diagnose the issue and construct the intended contrast, taking the target to be group 3 versus the average of groups 1 and 2.
Step 1: Check the coefficient sum. The proposed coefficients add to \(1-1+1=1\), not zero. Thus \(1\mu_1-\mu_2+\mu_3\) is not a contrast. It also does not represent group 3 versus the average of groups 1 and 2.
Step 2: Translate the stated target into coefficients. The target is \(\mu_3-(\mu_1+\mu_2)/2\), so the coefficients in group order are \((-1/2,-1/2,1)\). Their sum is \(-1/2-1/2+1=0\), as required.
Step 3: Estimate the contrast. Using the observed means gives \(\widehat L=20-(10+14)/2=20-12=8\) thousand cycles.
Check and common mistake. The estimate is exactly the difference between group 3's mean and the average of the other two means. A common mistake is to use \((1,-1,1)\) because it includes the same three groups; including the right groups does not make the coefficients represent the intended comparison. The corrected coefficients sum to zero and match the stated target.
Standard Errors: Use the Coefficients and the Sample Sizes
For a contrast estimated from independent group means, the variance depends on the square of each coefficient and the variance of that group's sample mean. Under the common-variance ANOVA model, the estimated standard error is
where \(n_i\) is the sample size in group \(i\). The corresponding test statistic for \(H_0:L=0\) is \(t=\widehat L/SE(\widehat L)\), with \(N-k\) error degrees of freedom, when the usual ANOVA assumptions hold. A confidence interval for \(L\) uses the same standard error and the appropriate \(t\) critical value.
Two recurring errors are to omit the square on a coefficient or to use a single sample size when group sizes differ. Neither is justified by the formula. If all groups have the same sample size, that common size can be factored out, but the squared coefficients must still be included.
Worked Example: Correcting a Contrast Standard Error
A hypothetical laboratory compares three curing protocols. The sample sizes are \(n_1=4\), \(n_2=6\), and \(n_3=5\), and the ANOVA gives \(MSE=9\) with \(15-3=12\) error degrees of freedom. The observed means are \(10\), \(12\), and \(16\), respectively. Test the single prespecified contrast \(L=\mu_3-(\mu_1+\mu_2)/2\) at the 5% two-sided level.
Step 1: Estimate the contrast. The coefficients are \((-1/2,-1/2,1)\), so \(\widehat L=16-(10+12)/2=16-11=5\).
Step 2: Calculate the standard error using each group size.
Step 3: Form the test statistic. \(t=5/1.6545\approx3.022\), with 12 degrees of freedom. The two-sided \(p\)-value is approximately \(0.01062\).
Step 4: State the conclusion. Since \(0.01062<0.05\), reject \(H_0:L=0\). The data provide evidence that the third protocol's population mean differs from the average of the first two.
Check and common mistake. The squared coefficients contribute \(1/4,1/4,1\), and the three group sizes remain distinct in the variance sum. Using \(MSE\) times an unsquared coefficient sum, or treating all three sample sizes as equal, would give the wrong standard error. This is a single prespecified contrast, so this example does not require an adjustment for a family of several tests.
Repeated Testing Requires a Defined Family
If several hypotheses are tested, the chance of at least one false rejection can exceed the nominal significance level for an individual test. The familywise error rate is the probability of making one or more Type I errors across a specified family of tests. A per-comparison level of \(0.05\) does not, by itself, ensure a familywise error rate of \(0.05\).
One general control is the Bonferroni procedure. For \(m\) tests and a desired familywise level \(\alpha\), test each hypothesis at level \(\alpha/m\). For two-sided \(t\) tests using a common ANOVA error term, the corresponding critical value is \(t_{1-\alpha/(2m),\,N-k}\). This procedure controls the familywise error rate even when the tests are dependent, although it can be conservative.
The family must be defined in a way that reflects the questions actually examined. For example, testing all three pairwise differences among three groups is a family of three comparisons. Calling only the most interesting result “the family” after testing all three does not undo the repeated testing.
Worked Example: Adjusting Three Prespecified Comparisons
A hypothetical study compares three training methods. It plans all three pairwise comparisons before data collection. The group sizes are \(9\), \(10\), and \(10\), the total sample size is \(29\), and \(MSE=16\). For one of the comparisons, the observed mean difference is \(4.4\). Use Bonferroni control at familywise level \(0.05\) to decide whether this comparison is significant.
Step 1: Identify the family and degrees of freedom. There are \(m=3\) planned pairwise comparisons. The ANOVA error degrees of freedom are \(29-3=26\).
Step 2: Compute the standard error for the stated pair. The two groups have sizes \(9\) and \(10\), so
Step 3: Calculate the test statistic. \(t=4.4/1.8379\approx2.394\).
Step 4: Apply the familywise adjustment. Each two-sided test uses level \(0.05/3=0.0166667\). The critical value is \(t_{1-0.05/(2\cdot3),26}\approx2.559\). Since \(2.394<2.559\), this comparison is not significant under the Bonferroni procedure.
Check and common mistake. The comparison would exceed the ordinary two-sided 5% critical value for 26 degrees of freedom, but the relevant question is whether it exceeds the adjusted critical value for the family of three. Reporting an unadjusted individual-test conclusion after examining all three would ignore the planned repeated testing.
Choosing Comparisons After Seeing the Means
A comparison selected after inspecting the sample means is not equivalent to a single comparison fixed in advance. If an analyst looks across several means and then tests the largest against the smallest, the selected difference tends to look unusually large partly because it was chosen for being extreme. An ordinary unadjusted test treats the pair as if it had been selected independently of the observed data and therefore does not account for that selection.
The appropriate approach depends on the inferential goal. If all pairwise differences are of interest, use a procedure designed for that family, such as Tukey or Tukey-Kramer under their applicable ANOVA conditions. A Bonferroni adjustment is another option. If the comparison is genuinely specified before examining the data, a single-comparison procedure can be appropriate. The key is not to select a favorable comparison and then report it as if it were the only planned test.
Worked Example: Adjusting for a Comparison Chosen from the Means
A hypothetical pilot study compares four formulations, with six independent observations per formulation. The observed means are \(10\), \(11\), \(12\), and \(13.2\), and the ANOVA error mean square is \(4\). After seeing the means, an analyst chooses the largest-versus-smallest difference. Use a Bonferroni family of all six pairwise comparisons at familywise level \(0.05\) to assess that selected difference.
Step 1: Define the family and error degrees of freedom. Four groups give \(4(4-1)/2=6\) pairwise comparisons. The total sample size is \(24\), so the error degrees of freedom are \(24-4=20\).
Step 2: Estimate the selected difference and its standard error. The observed largest-minus-smallest difference is \(13.2-10=3.2\). Both groups have size 6, hence
Step 3: Calculate the statistic and adjusted critical value. The statistic is \(t=3.2/1.1547\approx2.771\). With six two-sided comparisons, the Bonferroni critical value is \(t_{1-0.05/(2\cdot6),20}\approx2.927\).
Step 4: Conclude using the procedure that matches the selection. Since \(2.771<2.927\), the selected difference is not significant under this Bonferroni familywise procedure. An unadjusted single-comparison test would use a smaller critical value and could suggest significance, but it would fail to account for choosing the pair after inspecting all four means.
Check and common mistake. There are six possible pairs, not just the one eventually reported. The comparison's standard error is calculated from the two groups involved; the multiplicity adjustment reflects the larger set of comparisons from which that pair was selected.
A Practical Error-Checking Sequence
Before interpreting a contrast or comparison, check the question, the coefficients, the standard error, and the testing plan in that order. This separates errors in defining the estimand from errors in its uncertainty or in the decision rule.
Write the target as a linear combination of population means. For a contrast, verify that its coefficients sum to zero.
Apply each coefficient to the corresponding sample mean, in the same group order, and check that the result has the intended interpretation.
Use \(MSE\sum_i c_i^2/n_i\) for the estimated contrast variance, then take its square root. Use the ANOVA error degrees of freedom.
Distinguish one genuinely prespecified test from a family of comparisons, including comparisons selected after viewing the means.
Apply the chosen multiplicity procedure to the defined family and state the conclusion in context.
These checks are aligned with the SOA VEE Mathematical Statistics syllabus because they connect contrast definitions, pooled ANOVA uncertainty, and multiple-comparison decisions. The main discipline is to keep the inferential target and the testing procedure consistent from start to finish.
Check Your Understanding
For each question, check the coefficients, variance calculation, or testing family before deciding what the result means.
- A proposed comparison among four means uses coefficients \((1,1,-1,0)\). Is it a contrast? Give coefficients for group 3 versus the average of groups 1 and 2. (Answer: No; the proposed coefficients sum to \(1\). The intended contrast has coefficients \((-1/2,-1/2,1,0)\), which sum to zero.)
- Two independent groups have sample sizes \(5\) and \(10\), with \(MSE=25\). Find the ANOVA standard error for their mean difference. (Answer: \(\sqrt{25(1/5+1/10)}=\sqrt{7.5}\approx2.7386\).)
- Four pairwise hypotheses are tested using a common ANOVA error term with 18 error degrees of freedom. State the Bonferroni two-sided critical value for familywise level \(0.05\). (Answer: \(t_{1-0.05/(2\cdot4),18}=t_{0.99375,18}\).)
- Five group means are inspected and the largest-minus-smallest pair is then tested. How many pairwise comparisons are in the full family, and what Bonferroni two-sided per-test level controls familywise error at \(0.05\)? (Answer: \(5(4)/2=10\) comparisons; the per-test level is \(0.05/10=0.005\).)