Responding to Strong Nonnormality and Small Samples

Responding to Strong Nonnormality

A nonnormal-looking residual plot does not automatically invalidate an ANOVA, and a normality test cannot make the decision for you. The response should depend on the pattern of the residuals, the group sample sizes, the presence of unusual observations, and whether a scientifically sensible transformation can produce a more credible model. When those checks do not support a useful transformation, a rank-based alternative may be worth considering—but it answers a different kind of question.

The diagnostic target is the model error within groups, not simply the pooled raw outcomes. For a one-way ANOVA, write the model as $Y_{ij}=\mu_j+\varepsilon_{ij}$, where $Y_{ij}$ is observation $i$ in group $j$, $\mu_j$ is that group’s mean, and $\varepsilon_{ij}$ is its error. The usual model assumes independent errors with mean zero and a common variance; the classical small-sample $F$ inference also assumes normally distributed errors within each group. Residual plots provide evidence about whether that last assumption is plausible, but cannot prove it.

Key principle: Treat strong nonnormality as a prompt to investigate, not as an automatic command to transform or switch procedures. A good decision considers the residual pattern, sample size and balance, influential observations, the measurement scale, and the scientific quantity the analysis is meant to compare.

How Sample Size and Outliers Change the Decision

With larger samples, moderate departures from normality are often less consequential for mean comparisons, particularly when group sizes are similar and there are no severe outliers. This is a robustness consideration, not a guarantee: it does not repair dependence, unequal variances, or a handful of observations that dominate the means. In small samples, the data provide limited information about the shape of the underlying errors, and one unusual value can substantially affect a group mean and the residual pattern. There is no universal sample-size cutoff at which normality ceases to matter.

Balance matters too. A substantial difference in group sizes can make a comparison more vulnerable when group distributions are also differently variable or shaped. If unequal variances are the main concern, Welch ANOVA is an alternative for mean comparisons; nonnormality alone does not make it a rank procedure. The diagnostic question here is narrower: given the residual shape and the available sample, does a transformation give a defensible model, or is an analysis based on ranks more appropriate to consider?

Check what the residuals show.
Look for broad skewness, heavy tails, clusters, or a small number of extreme residuals. Consider whether these patterns occur within groups rather than arising only from differences between group means.

Consider sample size, balance, and unusual cases.
Ask how much influence each observation can have, whether groups are comparably sized, and whether an unusual value has a data, measurement, or scientific explanation.

Evaluate whether a transformation is credible.
A transformation is most persuasive when it suits the response scale and scientific context and improves the relevant residual diagnostics after the model is refitted.

Consider a rank-based analysis if the problem remains.
Check that its assumptions and interpretation match the study. Do not treat it as an automatic substitute for comparing means.

Worked Example: A Plausible Transformation With Moderate Skew

Suppose a hypothetical laboratory compares a positive concentration measured in three independent treatment groups, with 18 observations in each group. The raw-scale residual Q-Q plots show a right-skewed pattern in each group, but there is no isolated extreme residual. Residual spread also grows with the fitted level. The response is positive, and multiplicative changes in concentration are scientifically interpretable.

The equal group sizes reduce one source of vulnerability, but they do not erase the residual pattern. Nor does $n=18$ make the normal-error assumption irrelevant: the sample is not so large that a few influential observations can be dismissed, and the visible skew is consistent across groups rather than being caused just by different group means.

A logarithmic response is a reasonable candidate because concentrations are positive and multiplicative differences have a meaningful interpretation. The decision is not complete merely because a log transformation was applied. The group model must be refitted on the log scale, and its residual plots reconsidered. If the skew and changing spread are substantially reduced without creating a new pattern, the transformed analysis may be credible. If the same pronounced skew or influential residuals remain, the transformation has not resolved the concern. Any comparison then concerns means on the log scale unless results are subsequently expressed through an appropriate back-transformation; it is not automatically a comparison of raw-scale means.

When to Consider a Rank-Based Alternative

For independent groups, the Kruskal–Wallis procedure is a common rank-based alternative to the one-way ANOVA $F$ test. It replaces the observed responses with their pooled ranks, so a very large measurement does not contribute in proportion to its numerical distance from the other observations. That can be helpful when small samples and pronounced skew make a mean-based analysis especially sensitive to the response shape.

Definition: The Kruskal–Wallis statistic $H$ compares the rank sums across $k$ independent groups. Pool all $N$ responses and assign ranks from 1 to $N$; tied values receive their average rank. If group $j$ has $n_j$ observations and rank sum $R_j$, then, when there are no ties, $$ H=\frac{12}{N(N+1)}\sum_{j=1}^{k}\frac{R_j^2}{n_j}-3(N+1). $$ For sufficiently large samples, $H$ is commonly compared with a chi-square distribution with $k-1$ degrees of freedom. For very small samples, that approximation may be poor; an exact or permutation calibration, when justified by the design and null model, is preferable.

The procedure assumes independent observations and a response that can be meaningfully ordered. Its general null concerns the group distributions, not specifically equality of means. If the group distributions have comparable shapes and differ mainly by location, a significant result can be interpreted as evidence of a location difference, often described in terms of medians. If the shapes or spreads differ substantially, that simple median interpretation is not justified: a rank test can respond to distributional differences other than a shift in medians.

The statistic’s structure has a simple justification. Under a null in which the pooled ranks are exchangeable among groups, each group is expected to receive its share of the overall rank total. The average rank among $1,\ldots,N$ is $(N+1)/2$, so $$ E(R_j)=n_j\frac{N+1}{2}. $$ The Kruskal–Wallis statistic summarizes how far the observed rank sums depart from these expected rank sums, with a scaling that gives the usual large-sample reference distribution. This is why ranks, group sizes, and the null model—not the original measurement distances—determine the calculation.

Worked Example: Calculating a Kruskal–Wallis Statistic

Consider hypothetical independent observations from three groups, each with $n_j=3$: Group A has values 1, 2, and 8; Group B has values 3, 4, and 9; Group C has values 5, 6, and 10. There are no ties. The pooled ordered values are 1, 2, 3, 4, 5, 6, 8, 9, and 10, with ranks 1 through 9 in that order.

The rank sums are $R_A=1+2+7=10$, $R_B=3+4+8=15$, and $R_C=5+6+9=20$. With $N=9$, substitute into the no-ties formula:

\begin{aligned} H &=\frac{12}{9(10)} \left(\frac{10^2}{3}+\frac{15^2}{3}+\frac{20^2}{3}\right)-3(10)\\ &=\frac{12}{90}\left(\frac{100+225+400}{3}\right)-30\\ &=\frac{12}{90}\left(\frac{725}{3}\right)-30\\ &=2.2222\ldots \end{aligned}

The large-sample reference would use $df=k-1=2$, giving an approximate upper-tail p-value of about 0.329. But there are only three observations per group, so relying on that chi-square approximation is questionable. The calculation illustrates the statistic; it does not justify a confident conclusion from that approximation. For an actual analysis at this sample size, use an exact or permutation method if its assumptions are supported, and interpret any result as evidence about the group distributions unless comparable shapes make a location interpretation reasonable.

Normality Tests Are Evidence, Not a Verdict

A formal test of normality asks whether the observed data are sufficiently inconsistent with a specified normal model to reject it at a chosen significance level. Its p-value is not the probability that the errors are normal, and a large p-value does not establish normality. With a small sample, the test may have little power to detect important departures; with a large sample, it may flag departures that have little practical impact on the intended analysis. A Q-Q plot and the study context therefore remain important, alongside the test if one is used.

Worked Example: A Significant Test Does Not Choose the Analysis

In a hypothetical study, each of four independent groups has 40 observations. A normality test applied to the residuals gives $p=0.018$. The Q-Q plots show mild curvature in their tails, but no single extreme residual; group sizes are equal. This p-value is below a conventional $\alpha=0.05$, so the test rejects its normal-error null at that level. It does not, by itself, show that the ANOVA comparison is unusable or that a rank-based test is preferable.

The next step is to judge the size and pattern of the departure and its practical consequences. Here, moderate sample sizes, equal group sizes, and the absence of an influential point make it reasonable to assess the mean comparison’s robustness rather than switch procedures solely because of the p-value. The analyst should still inspect the residuals and relevant variance diagnostics, and consider the scientific target. If the tail pattern were severe or accompanied by influential observations, the evidence would call for a more substantial response.

Common Pitfalls and Practical Consequences

Letting a normality-test p-value make the decision. “The p-value is above 0.05, so the errors are normal” confuses failure to reject with proof. Conversely, rejection does not automatically invalidate every mean comparison. Use the test alongside residual plots, sample size, balance, and the consequences of the departure.
Ranking as if it still compared means. A rank procedure does not test equality of arithmetic means. State the distributional target, and use a median or location-shift interpretation only when the group shapes support it.
Transforming until a plot looks acceptable. A transformation should be compatible with the response and scientific question. After choosing one, refit the model and check its residuals; do not assume the original-scale interpretation is unchanged.
Deleting a genuine unusual observation. First check for recording or measurement error. If the observation is valid, examine its influence and report an analysis that reflects the data rather than removing it merely because it is inconvenient.
Assuming ranks solve every assumption problem. Kruskal–Wallis still requires independent observations and a suitable ordered response, and its result is not automatically a median comparison when group shapes differ.

These choices matter because the procedure determines what conclusion the data can support. If a transformed model is credible, it can preserve a comparison framed in terms of means on that transformed scale. A rank-based analysis may offer a useful response to severe shape problems, but changes the inferential target. An omnibus rank result also does not identify which particular groups differ; follow-up comparisons require an appropriate method that accounts for multiple testing.

Key takeaway: For strong nonnormality in small samples, verify unusual observations, assess how much the residual pattern threatens the intended mean comparison, and test whether a scientifically credible transformation improves the refitted model. Consider ranks when that route is not convincing, while stating clearly that the procedure addresses group distributions rather than automatically comparing means or medians.

Check Your Understanding

Use the assumptions and distinctions in this tutorial to answer the following.

Why should residual shape within groups be examined rather than judging normality only from the pooled raw outcomes?
Two groups are small, and one has a valid, very large observation. Why might that observation matter more than it would in a much larger sample?
A normality test has $p=0.20$. What does this result say, and what does it not establish?
What extra condition makes it reasonable to describe a Kruskal–Wallis result as evidence about group locations or medians?
In the worked rank example, explain why the value 8 receives rank 7 and how an incorrect rank would affect the resulting statistic.

Responding to Strong Nonnormality and Small Samples

What You'll Learn