Tutorials › Introduction to Statistics › Worked Workshop on Post Hoc Tests and Assumption Checks

Home  /  Tutorials  /  Multiple Comparisons and ANOVA Assumptions

Multiple Comparisons and ANOVA Assumptions · Tutorial 896 of 996

Worked Workshop on Post Hoc Tests and Assumption Checks

Learn how to connect residual checks to a suitable post hoc method, work through adjusted comparisons, and report conclusions that respect the assumptions.

Intermediate 11 min read

What You'll Learn

  • Check residual patterns, independence, and variance assumptions before choosing a post hoc procedure.
  • Calculate the pooled error estimate and Tukey simultaneous intervals for balanced groups.
  • Apply Holm adjustment to a family of pairwise p-values.
  • Use Welch ANOVA and Games–Howell comparisons when group variances are unequal.
  • Interpret adjusted results in context without confusing evidence about means with evidence about causes.

From Diagnostics to a Post Hoc Conclusion

A one-way ANOVA can indicate evidence that not all group means are equal, but it does not by itself identify which groups differ. Pairwise comparisons address that question while accounting for the fact that several comparisons are being made. The method should follow the design and the assumption checks: using a pooled-variance procedure when group variances are substantially unequal can give unreliable standard errors and conclusions.

This workshop follows a practical sequence: examine the design and ANOVA residuals, choose a method suited to the variance pattern, calculate or interpret adjusted comparisons, and state what the results support. Residual plots and tests provide evidence about model conditions; they do not prove that those conditions hold.

Model and conditions: In a one-way ANOVA with \(k\) groups, the observation in group \(i\) can be represented as \(Y_{ij}=\mu_i+\varepsilon_{ij}\), where \(\mu_i\) is that group’s population mean. The usual pooled-variance ANOVA and Tukey comparisons assume independent observations, approximately normal errors within each group, and a common error variance \(\sigma^2\). Independence is justified primarily by how observations were sampled or assigned. Residual checks help assess normality, equal variance, and unusual observations.

A residual is the observed value minus its fitted group mean: \(e_{ij}=Y_{ij}-\bar{x}_i\). For diagnostics, residuals are more informative than raw observations because they show deviations from the group means the model has fitted. A residual-versus-fitted plot can reveal changing spread or a pattern; a Q-Q plot can reveal substantial departures from normality. The design must still support independence, and potentially influential observations should be investigated rather than ignored.

Worked Example: Diagnostics and Tukey Comparisons

Worked Example: Diagnostics and Tukey Comparisons

Suppose a hypothetical experiment compares a measured response, in milligrams, across three independently sampled treatment groups. Each group has \(n_i=8\) observations. The group means are 12, 15, and 20 mg, and the sample standard deviations are 2.0, 2.5, and 2.2 mg. Assume the residual-versus-fitted plot shows no clear pattern or funnel, the residual Q-Q plot has no marked departure from a straight-line pattern, and there are no extreme residuals. The study design supports independent observations. The group standard deviations are reasonably similar, so a pooled-variance analysis is a defensible choice for this example.

First calculate the pooled error variance, \(MSE\), using the within-group sums of squares:

$$ MSE=\frac{\sum_{i=1}^{3}(n_i-1)s_i^2}{N-k} =\frac{7(2.0^2)+7(2.5^2)+7(2.2^2)}{24-3} =\frac{105.63}{21}=5.030 $$

The grand mean is \((12+15+20)/3=15.667\) mg because the group sizes are equal. The between-group sum of squares is \(8[(12-15.667)^2+(15-15.667)^2+(20-15.667)^2]=261.333\). Thus \(MS_{\text{between}}=261.333/2=130.667\), and the ANOVA statistic is \(F=130.667/5.030=25.98\), with 2 and 21 degrees of freedom. This gives strong evidence against the null hypothesis that all three population means are equal (\(p<0.001\)).

For equal group sizes, Tukey’s method compares every pair using the pooled error estimate. Here the standard error scale for a difference is \(\sqrt{MSE/n}=\sqrt{5.030/8}=0.7929\) mg. For three groups and 21 error degrees of freedom, the 5% studentized-range critical value is approximately \(q^*=3.56\). The simultaneous margin is \(3.56(0.7929)=2.82\) mg. Therefore each estimated difference receives an interval with this margin:

$$ \begin{aligned} \mu_2-\mu_1 &: (15-12)\pm2.82=(0.18,\ 5.82)\text{ mg},\\ \mu_3-\mu_1 &: (20-12)\pm2.82=(5.18,\ 10.82)\text{ mg},\\ \mu_3-\mu_2 &: (20-15)\pm2.82=(2.18,\ 7.82)\text{ mg}. \end{aligned} $$

All three intervals exclude zero. At the familywise 5% level, these data support differences in mean response for all three pairs. The intervals concern population mean differences; they do not imply that every observation in one group exceeds every observation in another.

Why Tukey’s Simultaneous Coverage Works

A common source of confusion is to treat the observed range of sample means as though it had the central studentized-range distribution whenever the means differ. That is not generally true: under unequal population means, the observed range is shifted by the differences among those means. The coverage argument instead uses the range of the standardized estimation errors, which has the central studentized-range distribution under the model.

For equal group sizes, write each sample mean as \(\bar{x}_i=\mu_i+\varepsilon_i\). Under independent normal errors with common variance, the standardized errors \(\varepsilon_i/\sqrt{MSE/n}\), considered together with the pooled estimate of error, have a studentized-range statistic. Choosing \(q^*\) as its \(1-\alpha\) quantile gives:

$$ P\left( \frac{\max_i\varepsilon_i-\min_i\varepsilon_i}{\sqrt{MSE/n}} \leq q^* \right)=1-\alpha $$

On the event inside this probability, every pairwise error difference is bounded in absolute value: \(|\varepsilon_i-\varepsilon_j|\leq q^*\sqrt{MSE/n}\). Since \((\bar{x}_i-\bar{x}_j)-(\mu_i-\mu_j)=\varepsilon_i-\varepsilon_j\), every interval \((\bar{x}_i-\bar{x}_j)\pm q^*\sqrt{MSE/n}\) then covers its corresponding population difference. The same error-range event protects all pairs at once, giving simultaneous coverage \(1-\alpha\). Tukey–Kramer extends this idea to unequal group sizes by using the pair-specific scale \(\sqrt{MSE(1/n_i+1/n_j)/2}\) with the studentized-range critical value.

When Holm Is a Suitable Choice

Tukey is designed for all pairwise comparisons under the pooled-variance model. Holm’s method is another way to control the familywise error rate—the probability of making one or more false rejections in a family of tests. It adjusts a set of valid p-values by ordering them and testing the smallest first. Under equal variances, the starting pairwise tests can be pooled-variance \(t\)-tests. Holm can also be used with other valid individual tests; its adjustment does not repair invalid standard errors or an unsuitable model.

Worked Example: Applying Holm’s Adjustment

Suppose four groups are compared pairwise, giving six raw p-values from suitable pooled-variance tests: 0.001, 0.008, 0.014, 0.031, 0.044, and 0.200. Assume the independence, approximate normality, and common-variance conditions have been checked as in the pooled model, and use familywise \(\alpha=0.05\).

Order the p-values from smallest to largest. Holm compares the first to \(0.05/6=0.00833\), the second to \(0.05/5=0.0100\), and the third to \(0.05/4=0.0125\). The first two pass their thresholds, but the third does not: \(0.014>0.0125\). The step-down procedure stops there, so only the first two comparisons are rejected.

The corresponding Holm-adjusted p-values, in the original sorted order, are the running maxima of \((6-j+1)p_{(j)}\), capped at 1. The products are 0.006, 0.040, 0.056, 0.093, 0.088, and 0.200; taking running maxima gives 0.006, 0.040, 0.056, 0.093, 0.093, and 0.200. Thus only the first two adjusted p-values are at most 0.05. Report which pairs those p-values correspond to, rather than merely saying that “some groups differ.”

When Variances Differ: Welch and Games–Howell

If residual spread differs substantially among groups, pooling the variances can misrepresent uncertainty, especially when group sizes are also unequal. Welch’s one-way ANOVA does not assume a common variance. When it is appropriate to follow up with all pairwise comparisons, Games–Howell uses pair-specific standard errors and degrees of freedom, with a studentized-range adjustment for the family of comparisons. Both methods still require independent observations; Welch and Games–Howell do not make dependence or severe nonnormality disappear.

Welch ANOVA and Games–Howell: For group \(i\), let \(w_i=n_i/s_i^2\), and let \(\bar{x}_w=\sum_iw_i\bar{x}_i/\sum_iw_i\). Welch’s statistic weights the means using these inverse-variance weights and uses an adjusted denominator and degrees of freedom. For a Games–Howell comparison of groups \(i\) and \(j\), the estimated difference is \(\bar{x}_i-\bar{x}_j\), and its standard error is \(\sqrt{s_i^2/n_i+s_j^2/n_j}\). The comparison uses a pair-specific Welch–Satterthwaite degrees of freedom and a studentized-range adjustment.

Worked Example: Unequal Variances and Games–Howell

Consider hypothetical independent groups with \(n_i=10\) each, means of 10, 14, and 19 mg, and standard deviations of 1, 2, and 3 mg. The spread increases noticeably across groups, so the equal-variance model is not a good choice; use Welch ANOVA and Games–Howell. The Welch weights are \(10/1^2=10\), \(10/2^2=2.5\), and \(10/3^2=1.111\), with total weight \(13.611\). The weighted mean is \((10(10)+2.5(14)+1.111(19))/13.611=11.469\) mg.

For three groups, Welch’s statistic can be written as \(F_W=[\sum_i w_i(\bar{x}_i-\bar{x}_w)^2/(k-1)]/C\), where \(C=1+[2(k-2)/(k^2-1)]\sum_i(1-w_i/\sum w_i)^2/(n_i-1)\). Here the weighted sum in the numerator is approximately \(100.61\), so its value after division by \(k-1=2\) is \(50.305\). The correction sum is approximately \(0.1755\), giving \(C=1+(2/8)(0.1755)=1.0439\). Thus \(F_W=50.305/1.0439=48.19\). The denominator degrees of freedom are approximately \((k^2-1)/(3\sum_i(1-w_i/\sum w_i)^2/(n_i-1))=15.19\). The result provides strong evidence that the population means are not all equal (\(p<0.001\)).

For the comparison of groups 1 and 2, the standard error is \(\sqrt{1^2/10+2^2/10}=\sqrt{0.5}=0.7071\) mg. The Welch–Satterthwaite degrees of freedom are \(0.5^2/[(0.1^2/9)+(0.4^2/9)]=13.24\). Its studentized-range statistic is \(\sqrt{2}(14-10)/0.7071=8.00\). The other estimated differences are 9 mg and 5 mg; their standard errors are \(\sqrt{1.0}=1.000\) mg and \(\sqrt{1.3}=1.140\) mg, respectively. Their studentized-range statistics are approximately 12.73 and 6.20. Using the appropriate pair-specific degrees of freedom, the 5% Games–Howell critical values are roughly 3.8. Each statistic exceeds its critical value, so all three adjusted comparisons are significant at the familywise 5% level. The estimated mean differences are 4, 9, and 5 mg, respectively.

For example, a Games–Howell interval for groups 1 and 2 uses the form \((\bar{x}_2-\bar{x}_1)\pm(q^*/\sqrt{2})SE\). With a critical value near 3.8, this is approximately \(4\pm(3.8/\sqrt{2})(0.7071)\), or \((2.10,5.90)\) mg. A software output should be consulted for the precise pair-specific critical value, interval, and adjusted p-value.

Turning the Analysis into a Clear Report

A concise report names the method, identifies the assumption checks relevant to that method, and gives the adjusted pairwise conclusions in the response’s units. Distinguish the overall ANOVA result from the follow-up comparisons: a significant overall test says that the means are not all equal, while adjusted pairwise results identify the pairs for which the data support differences. Neither result establishes that group membership caused the differences unless the study design supports a causal interpretation.

1
Check the design and residuals.
Use the study design to assess independence; inspect residual-versus-fitted and Q-Q plots and check for influential observations.
2
Choose the variance approach.
When a common variance is reasonable, use a pooled ANOVA and a suitable familywise procedure such as Tukey or Holm. When variances differ substantially, consider Welch ANOVA followed by Games–Howell comparisons.
3
Report adjusted results.
Name the pairs, give estimated differences and adjusted uncertainty or p-values, state the familywise level, and express differences in context and units.

Common Mistakes

  • Choosing a procedure before checking the model. A familiar post hoc method is not automatically suitable. Check design and residual evidence first; use a variance-robust alternative when the common-variance assumption is not defensible.
  • Treating the observed range of sample means as the coverage argument. The central studentized-range distribution describes the standardized range of estimation errors under the model, not generally the observed range when population means differ. Tukey’s simultaneous coverage follows because bounding the error range bounds every pairwise estimation error.
  • Scaling Welch weights without adjusting the numerator. The weighted mean is unchanged if all weights are divided by the same constant, but the weighted sum of squared deviations is scaled too. Use the original weights in the stated formula, or undo the scaling before calculating the statistic.
  • Reading an overall significant test as proof that every pair differs. Use adjusted comparisons to identify supported pairwise differences and report their uncertainty.
  • Ignoring the family of tests. Reporting unadjusted pairwise p-values as though only one comparison had been planned can inflate the chance of at least one false rejection. Use an appropriate adjustment and state it.
Key takeaway: A sound post hoc conclusion depends on a chain of decisions: assess the design and residuals, select a method consistent with the variance pattern, adjust for the comparisons being made, and report pairwise findings in context.

Check Your Understanding

Use the reasoning in the workshop to answer these questions.

  1. Why does a residual-versus-fitted plot help assess equal variance more directly than a plot of the raw observations against group labels?
  2. In the balanced three-group example, what does the Tukey interval from 0.18 to 5.82 mg say about the population mean difference?
  3. Why is the central studentized-range distribution used to justify simultaneous coverage, rather than asserted to describe the observed range of sample means for any values of the population means?
  4. In Holm’s procedure, what happens after an ordered p-value fails its corresponding sequential threshold?
  5. Why are Welch ANOVA and Games–Howell preferable to pooled-variance procedures when group variances differ substantially?

Previous

Next

Back to Tutorials