Why Careful Interpretation Matters
Post hoc comparisons and assumption checks can make an ANOVA analysis more informative, but they also create opportunities for overconfident conclusions. Testing many pairs without adjustment can produce false positives. Choosing which pairs to report after inspecting their results can make the reported evidence look stronger than it is. And a diagnostic test cannot certify that a model assumption is true.
The central discipline is to keep the analysis question in view. Define which comparisons belong to the inferential family, use a method suited to that family, and treat diagnostics as imperfect evidence about the model. A small p-value, a large p-value, or a tidy-looking plot does not replace these decisions.
Unadjusted Pairwise Tests Do Not Control the Familywise Error Rate
Suppose an ANOVA compares \(k\) group means. If every pair is compared, there are \(m=k(k-1)/2\) pairwise hypotheses. Testing each one at significance level \(\alpha\) does not generally keep the probability of at least one false rejection across the whole family at \(\alpha\). That probability is called the familywise Type I error rate.
For intuition, if \(m\) tests were independent and all their null hypotheses were true, each test would avoid a false rejection with probability \(1-\alpha\). The probability of avoiding false rejections on all tests would then be \((1-\alpha)^m\). Therefore,
This derivation uses independence, which pairwise tests often do not satisfy. It still shows why testing more hypotheses creates more opportunity for a false positive. More generally, the Bonferroni inequality gives an upper bound without requiring independence: if each of \(m\) tests uses level \(\alpha/m\), the probability of one or more false rejections is at most \(\alpha\).
Different procedures control error in different ways and suit different comparison goals. For example, all-pairs comparisons and comparisons against a control are different families. The procedure should match the scientific question, rather than being chosen because it produces the smallest p-values.
Worked Example: An Unadjusted Pairwise Result
A hypothetical experiment compares the mean recovery time for three treatments. The analysis includes all three pairwise comparisons. One unadjusted pairwise test reports \(p=0.018\). Use a Bonferroni adjustment to assess this result at familywise \(\alpha=0.05\).
There are \(m=3(3-1)/2=3\) comparisons. The Bonferroni per-comparison level is \(\alpha/m=0.05/3=0.0167\), to three significant figures. Since \(0.018>0.0167\), this result does not meet the Bonferroni-adjusted criterion.
Equivalently, the Bonferroni-adjusted p-value is \(\min(1,mp)=\min(1,3(0.018))=0.054\). At familywise level 0.05, the data do not provide sufficient adjusted evidence of a difference in mean recovery time for this pair. This does not establish that the means are equal; it reports the conclusion under the specified error control.
Define the Comparison Family Before Looking at Results
A comparison family is the set of hypotheses for which a joint error rate is being controlled. It should reflect the claims the analysis is intended to support. If the scientific question concerns every pair of groups, the family consists of all pairwise comparisons. If the question concerns each treatment versus a control, the family consists of those treatment-control comparisons.
A common mistake is to inspect the results first, then report only the comparisons that look interesting and describe them as if they had been selected in advance. This selection can exaggerate evidence: among many tests, the smallest p-values are especially likely to look compelling by chance. An adjustment cannot be justified merely by counting the comparisons left in the final report if the analyst searched a larger set and selected results based on their outcomes.
This does not mean every exploratory analysis is invalid. It means the interpretation must reflect how the comparisons were chosen. State whether comparisons were planned or selected after examining the data, identify the relevant family, and use an adjustment appropriate to that family. If the analysis has several distinct families of claims, their error-control goals should be made clear rather than silently treating them as one.
Worked Example: Selecting the Smallest P-Value
A hypothetical study compares four groups. After seeing all results, an analyst highlights the pair with the smallest unadjusted p-value, \(p=0.012\), and reports it as significant at 0.05. Suppose the intended claim is that any two groups may differ, so all pairwise comparisons are relevant.
There are \(m=4(4-1)/2=6\) pairs. A Bonferroni calculation for that family gives a per-comparison level of \(0.05/6=0.00833\). The selected result, \(0.012\), exceeds this threshold. Its Bonferroni-adjusted p-value is \(\min(1,6(0.012))=0.072\), so it does not meet the familywise 0.05 criterion.
The problem is not solved by saying that only one comparison appears in the report: the analyst searched six comparisons and selected the smallest result. If the research question had instead specified treatment-versus-control comparisons in advance, the family and suitable procedure could be different. The scientific question, not the observed p-values, should determine that choice.
ANOVA Normality Concerns Residuals, Not Pooled Raw Values
In a one-way ANOVA model, an observation in group \(i\) can be represented as \(Y_{ij}=\mu_i+\varepsilon_{ij}\), where \(\mu_i\) is the mean for group \(i\) and \(\varepsilon_{ij}\) is the random deviation from that group mean. The classical model assumes independent errors with a common variance; the normality condition concerns the errors within groups, not a single distribution formed by pooling observations from groups with different means.
A residual is an estimated error, commonly calculated as \(e_{ij}=y_{ij}-\bar{x}_i\), where \(y_{ij}\) is an observed response and \(\bar{x}_i\) is its group sample mean. Residual plots and Q-Q plots are useful for examining whether the model's error pattern is plausible. They do not turn a sample into a definitive test of the underlying error distribution.
Worked Example: Why a Raw-Data Normality Test Can Mislead
Consider three hypothetical groups whose underlying response distributions are normal with standard deviation 1, but whose means are 10, 20, and 30 units. The pooled raw observations combine three distributions centered far apart. A normality test applied to all raw observations together may reject a single normal distribution.
That rejection does not by itself show that the ANOVA normality condition fails. The condition concerns the distributions around their respective group means. For illustration, if observed group values were 9, 10, 11; 19, 20, 21; and 29, 30, 31, their sample means would be 10, 20, and 30. Subtracting each group's mean gives residuals \(-1,0,1\) in each group. The pooled raw values have large separations between groups, while the residuals represent within-group deviations.
These nine illustrative observations are too few to establish normality, and their residuals do not prove that a normal model is appropriate. The example shows why testing pooled raw values answers the wrong question when group means differ. Examine residuals in the context of the model, along with the study design and sample sizes.
Assumption Tests Are Evidence, Not Proof
A formal test for an assumption gives evidence against a specified null model; it cannot verify that the model is true. A small p-value from a normality or equal-variance test indicates that the observed data would be unusual under that test's null model. A large p-value means the test did not detect sufficiently strong evidence against that model. It does not mean the assumption has been proved.
The strength of a diagnostic test depends partly on sample size and the nature of the departure. With small samples, important departures may go undetected. With very large samples, a test can flag minor departures that have little practical consequence for the analysis. Graphs can reveal patterns that a single test statistic conceals, but graphs also have limited power, especially with few observations.
Normality and equal variance are not the same assumption, and a check aimed at one does not settle the other. Residual plots, Q-Q plots, and formal tests should be interpreted for the specific feature they address. Independence is different again: it is primarily justified from how observations were sampled or assigned, not established by a normality test or a residual plot.
Worked Example: Interpreting a Nonsignificant Assumption Test
In a hypothetical ANOVA, a formal equal-variance test returns \(p=0.31\) at \(\alpha=0.05\). The group sizes are small, and the residual plot shows some possible increase in spread at larger fitted values. What can be concluded?
Because \(0.31>0.05\), the test does not provide sufficient evidence to reject its equal-variance null hypothesis at this level. It does not show that the group variances are equal. Small group sizes may make departures difficult to detect, and the residual plot raises a concern worth investigating.
A careful report would describe both pieces of evidence and consider whether the planned ANOVA procedure is suitable given the observed pattern and design. It would not write “equal variances were proved” or use the nonsignificant p-value as a guarantee. If unequal variances are a substantial concern, a method designed for that situation may be more appropriate.
A Practical Sequence for Avoiding These Errors
A disciplined analysis keeps decisions and evidence in the right order. The following sequence is a useful safeguard:
Specify which group differences matter and whether they were planned before examining the outcome data.
Identify the set of claims for error control and choose a multiple-comparison method suited to those claims.
Use residual diagnostics for within-group model behavior, formal tests as supporting evidence, and study design to assess independence.
Give the adjustment, the relevant family, and the diagnostic evidence without claiming that a nonsignificant test proves an assumption.
Common Mistakes
- Reporting many unadjusted pairwise p-values as separate 0.05 tests. The correction is to account for the comparison family when controlling the chance of any false rejection.
- Defining the family after seeing which comparisons are significant. The correction is to base the family on the scientific question and disclose data-driven selection.
- Testing pooled raw observations for ANOVA residual normality. Different group means can make the pooled distribution nonnormal even when within-group errors are normal; assess residuals for the fitted model instead.
- Treating a large assumption-test p-value as proof. It indicates insufficient evidence against the test's null model, not confirmation that the assumption holds.
- Using diagnostic plots to claim independence. Independence is mainly supported by the sampling and assignment design; it cannot generally be established from a plot alone.
Check Your Understanding
Try these questions without looking back.
- For five groups, how many all-pairs comparisons are there, and why can testing each at 0.05 raise the familywise error rate?
- An analyst searches eight comparisons and reports only the smallest unadjusted p-value. Why is it not enough to count just the reported comparison when assessing selection?
- Why might a normality test on pooled raw observations be misleading in an ANOVA with substantially different group means?
- A formal assumption test has \(p=0.42\). What does this result say, and what does it not establish?
- Which source of evidence is especially important when evaluating independence, and why can a residual plot not prove independence?