Normality and Equal-Variance Diagnostics

Why Diagnose the One-Way ANOVA Assumptions?

A one-way fixed-effects ANOVA compares several group means using variation within groups to represent random error. Its usual model assumes independent observations, normally distributed errors, and a common error variance across groups. Independence is principally justified by the sampling design; normality and equal variance can also be examined using the observed data.

Diagnostics do not prove that the model is correct or incorrect. They help identify patterns that make the model less plausible and guide how cautiously to interpret an ANOVA result. A plot that looks irregular with only a few observations per group is a warning to investigate, not a conclusive test. Conversely, a tidy plot cannot guarantee that the assumptions hold.

Model and diagnostic quantities: In a one-way fixed-effects model, write $Y_{ij}=\mu_i+\varepsilon_{ij}$, where $Y_{ij}$ is observation $j$ in group $i$, and $\mu_i$ is that group’s mean. The usual model assumes that the errors $\varepsilon_{ij}$ are independent, have mean zero, have common variance $\sigma^2$, and are normally distributed. The observed residual is $e_{ij}=Y_{ij}-\bar{Y}_i$, where $\bar{Y}_i$ is the sample mean in group $i$. Errors are unobserved; residuals are calculated from the data and are estimates of those errors.

What to Look For in Residual Plots

A residual plot places residuals on the vertical axis and uses group labels, fitted group means, or another relevant ordering on the horizontal axis. In a group-specific plot, examine the distribution of residuals within each group rather than pooling the original observations. The group means may legitimately differ; subtracting each group’s own mean removes those differences before assessing the within-group patterns.

For a well-behaved model, residuals in each group should be centered around zero, with no systematic shape or unexplained clustering. A curved pattern, one-sided tail, or isolated point can suggest a departure worth examining. If the horizontal axis is simply the group label, the plot can also make differences in spread visible. If observations have a meaningful order, plotting residuals in that order may reveal trends or runs; such patterns may indicate dependence or another design issue, not necessarily a normality problem.

Worked Example: Comparing Residual Spreads Across Three Groups

A hypothetical laboratory compares a measurement across three treatment groups. The observations are $A: 8,10,12$, $B: 18,20,22$, and $C: 27,30,33$. Calculate each group’s residuals and sample standard deviation, then assess the evidence about common variance.

Method. Subtract each group’s own mean to obtain its residuals. Calculate each sample variance using the sum of squared residuals divided by $n_i-1$, where $n_i=3$ here.

For group A, the mean is $(8+10+12)/3=10$, so the residuals are $-2,0,2$. Their squared sum is $4+0+4=8$, giving $s_A^2=8/2=4$ and $s_A=2$.

For group B, the mean is $(18+20+22)/3=20$, so the residuals are $-2,0,2$. Their squared sum is again $8$, giving $s_B^2=8/2=4$ and $s_B=2$.

For group C, the mean is $(27+30+33)/3=30$, so the residuals are $-3,0,3$. Their squared sum is $9+0+9=18$, giving $s_C^2=18/2=9$ and $s_C=3$.

The observed standard deviations are 2, 2, and 3, and the ranges are 4, 4, and 6. Group C is more spread out in this sample, but each group contains only three observations. This is a reason to inspect further, not enough evidence by itself to declare that the population variances differ.

Check and trap avoided. Each group’s residuals sum to zero, as they must when calculated from its own mean. Comparing the original measurements’ overall spread instead would mix differences between group means with within-group variation.

Normal Quantile Plots for Within-Group Shape

A normal quantile plot (also called a normal probability plot or Q–Q plot) compares ordered observations with the ordered quantiles expected from a normal distribution. For ANOVA diagnostics, make or inspect such plots for residuals within each group, or examine a plot of all residuals while remembering that pooling can conceal differences between groups. Points that fall roughly along a straight line are compatible with normality; systematic curvature can suggest skewness, and an extreme point far from the overall pattern can suggest an outlier or a heavy tail.

The plot is a visual comparison, not a pass-or-fail rule. With a small group, there are few points to reveal a pattern, and one unusual observation can strongly affect the appearance. With larger samples, small departures may become visible even when they have little practical consequence. Always consider the plot alongside the group’s size, spread, and context.

Worked Example: Spotting a Possible Right-Tail Departure

Three hypothetical groups produce measurements $A: 2,3,3,4,8$, $B: 12,13,14,15,16$, and $C: 22,23,24,25,26$. Calculate the within-group residuals, identify what a normal quantile plot of group A would flag, and consider the spread evidence.

Method. Center the observations separately within each group, then inspect the ordered residuals for a departure from the roughly symmetric pattern expected under normality.

Group A has mean $(2+3+3+4+8)/5=20/5=4$. Its residuals are $-2,-1,-1,0,4$, already in increasing order. The last residual, 4, is separated from the others on the positive side. A normal quantile plot of these five residuals would therefore merit attention for possible right-skewness or an unusual high observation.

Group B has mean $70/5=14$, giving residuals $-2,-1,0,1,2$. Group C has mean $120/5=24$, also giving residuals $-2,-1,0,1,2$. Those two groups have symmetric residual patterns in this small sample.

For group A, the squared residuals sum to $4+1+1+0+16=22$, so $s_A^2=22/4=5.5$, and $s_A\approx2.345$. For groups B and C, the squared residuals sum to $4+1+0+1+4=10$, so each has $s^2=10/4=2.5$ and $s\approx1.581$. Group A’s larger spread is consistent with the high residual contributing substantially to its sum of squares.

Conclusion and trap avoided. The high measurement in group A is a diagnostic signal, not proof that the observation is erroneous or that the errors are nonnormal. Do not delete it solely to improve the plot; check the measurement and study context, and report the limited evidence from only five observations.

Assessing the Common-Variance Assumption

The common-variance assumption means that the error variance is the same in every group. A practical first comparison is among group-specific standard deviations, supplemented by boxplots or interquartile ranges. Standard deviations use all observations but can be sensitive to extreme values; interquartile ranges are less sensitive to extremes but describe only the middle half of the data. A side-by-side view of group-specific residuals can help distinguish a difference in spread from a difference in group location.

There is no universal ratio of sample spreads that, by itself, proves equal or unequal population variances. Interpret a difference in the setting of group sizes, outliers, and plot patterns. Unequal group sizes deserve particular care: a substantial variance difference can matter more when sample sizes also differ, but diagnostics alone do not determine the consequences for an ANOVA analysis.

Worked Example: Quantifying a Difference in Group Spreads

A hypothetical materials study records these values for three formulations: $A: 9,10,10,11$, $B: 17,18,22,23$, and $C: 28,29,31,32$. Calculate each sample standard deviation and compare the largest with the smallest.

Method. Compute each group mean, sum the squared deviations from that group mean, divide by $n_i-1=3$, and take the square root.

Group A has mean $(9+10+10+11)/4=40/4=10$. Its deviations are $-1,0,0,1$, whose squared sum is $2$. Thus $s_A^2=2/3$ and $s_A=\sqrt{2/3}\approx0.8165$.

Group B has mean $(17+18+22+23)/4=80/4=20$. Its deviations are $-3,-2,2,3$, whose squared sum is $9+4+4+9=26$. Thus $s_B^2=26/3$ and $s_B=\sqrt{26/3}\approx2.9439$.

Group C has mean $(28+29+31+32)/4=120/4=30$. Its deviations are $-2,-1,1,2$, whose squared sum is $4+1+1+4=10$. Thus $s_C^2=10/3$ and $s_C=\sqrt{10/3}\approx1.8257$.

The largest-to-smallest standard-deviation ratio is $2.9439/0.8165\approx3.6$, to one decimal place. The observed spreads are notably different and warrant careful review of group-specific plots and the data collection. They do not, on their own, establish the population variance relationship.

Check and trap avoided. The means are 10, 20, and 30, but those location differences are not evidence of unequal error variance. Calculating deviations from each group’s own mean keeps the spread comparison focused on within-group variability.

Why Residuals Need Careful Interpretation

Residuals are not identical to the unobserved errors: each residual subtracts an estimated group mean. In fact, within every group their sum is zero, so they cannot be treated as independent observations. This also explains why raw measurements pooled across groups are the wrong quantities for checking the shape and spread of the errors: pooled measurements retain the differences in group means.

Justification: Under the model, $e_{ij}=\varepsilon_{ij}-\bar{\varepsilon}_i$, where $\bar{\varepsilon}_i$ is the average error in group $i$. If the errors are independent with variance $\sigma^2$, then $\operatorname{Var}(\bar{\varepsilon}_i)=\sigma^2/n_i$ and $\operatorname{Cov}(\varepsilon_{ij},\bar{\varepsilon}_i)=\sigma^2/n_i$. Therefore, $$ \operatorname{Var}(e_{ij}) =\operatorname{Var}(\varepsilon_{ij})+\operatorname{Var}(\bar{\varepsilon}_i)-2\operatorname{Cov}(\varepsilon_{ij},\bar{\varepsilon}_i) =\sigma^2\left(1-\frac{1}{n_i}\right). $$ Also, $\sum_j e_{ij}=\sum_j(Y_{ij}-\bar{Y}_i)=0$, by the definition of the group mean. Thus residuals are useful indicators of error behavior, but their dependence and their adjustment for estimated means matter when interpreting plots.

A Practical Diagnostic Sequence

Respect the design.
Use the sampling design to assess whether independence is plausible; a residual plot cannot repair dependence caused by the way observations were collected.

Center within groups.
Calculate or display $e_{ij}=Y_{ij}-\bar{Y}_i$, rather than judging the normality or spread of uncentered measurements across groups.

Inspect shape and spread.
Use group-specific residual plots and normal quantile plots for possible shape departures; compare group standard deviations and boxplots for spread differences.

Investigate, then qualify.
Check unusual observations and the study context. State what the diagnostics suggest and how limited the evidence is, rather than declaring an assumption proven.

Common Mistakes and Exam Traps

Using the overall spread of raw measurements. If groups have different means, raw pooled spread includes both between-group location differences and within-group variation. For example, the nine observations in the first worked example have overall mean 20; their squared deviations from 20 sum to 634, giving a sample variance of $634/8=79.25$. The within-group squared deviations instead sum to $8+8+18=34$, across $9-3=6$ within-group degrees of freedom, giving $34/6\approx5.667$. These quantities answer different questions.
Calling a normal quantile plot a formal verdict. A roughly straight pattern is compatible with normality, not proof of it. A small sample can hide departures or make one observation dominate.
Pooling residuals without checking each group. Different group shapes can be obscured by pooling. Compare group-specific patterns as well as any overall display.
Treating unequal sample standard deviations as proof of unequal population variances. Sample spreads vary by chance, especially in small groups. Describe the evidence and consider group sizes and unusual values.
Confusing a mean difference with a variance difference. The ANOVA groups may have different means under the model. Center measurements within groups before examining error patterns.

Key takeaway: Normal quantile plots and group-specific residual and spread comparisons are diagnostic evidence, not definitive tests. Center within groups, interpret patterns in context, and distinguish observed sample behavior from conclusions about the underlying error distributions.

Check Your Understanding

For each question, use group-centered residuals when assessing within-group shape or spread.

A group has observations $5,7,9$. Find its mean and residuals. (Answer: Mean $7$; residuals $-2,0,2$.)
Two groups have sample standard deviations 1.5 and 3.0, with six observations in each. What is the largest-to-smallest standard-deviation ratio, and does it prove unequal population variances? (Answer: Ratio $2.0$; no, it is a descriptive warning, not proof.)
A normal quantile plot of one group’s residuals is nearly straight except for one high point. Give one reasonable interpretation and one conclusion that is not justified. (Answer: The point may indicate an unusual observation or a possible upper-tail departure; it does not by itself prove nonnormality or justify deleting the observation.)
Group A has values $3,4,5$, and group B has values $13,14,15$. What are their residual sequences, and why should the ten-unit difference between their means not be treated as evidence of unequal error variance? (Answer: Both residual sequences are $-1,0,1$; the mean difference is a location difference, while equal-variance diagnostics compare within-group spreads.)
A group has residuals $-2,-1,1,2$. Calculate its sample variance and standard deviation. (Answer: Squared deviations sum to $4+1+1+4=10$; sample variance $10/3$, standard deviation $\sqrt{10/3}\approx1.826$.)

What You'll Learn