Finding Outlying and Influential ANOVA Residuals

Why Look Beyond the ANOVA F Statistic?

An ANOVA result summarizes differences among group means relative to variation within groups. But a single observation can be unusual compared with the other observations in its group, or can have a disproportionate effect on that group's mean. Those are related but distinct concerns: an observation may have a large residual without substantially changing a mean, or may affect a mean enough to alter the analysis. Diagnostics help identify these cases; they do not, by themselves, justify removing data.

The one-way ANOVA model assumes independent errors, normally distributed errors within each group, and a common error variance across groups. In symbols, an observation in group \(i\) is modeled as \(Y_{ij}=\mu_i+\varepsilon_{ij}\), where the errors have mean 0 and variance \(\sigma^2\). These assumptions are about the errors around the group means, so residuals are the relevant quantities to inspect. Independence must be judged from the study design, not established by a residual plot.

Residuals and Studentized Diagnostics

Let \(n_i\) be the number of observations in group \(i\), \(N\) the total number of observations, and \(k\) the number of groups. The within-group sum of squares is \(SSE=\sum_i\sum_j e_{ij}^2\), and the pooled estimate of error variance is \(MSE=SSE/(N-k)\). The standard deviation of a raw residual is not simply \(\sqrt{MSE}\): fitting the group mean constrains the residuals within each group to sum to zero. In fact, the residual variance for an observation in group \(i\) is \(\sigma^2(1-1/n_i)\).

Dividing a residual by an estimated standard deviation puts it on a scale that is easier to compare across observations. An internally studentized residual uses the full-data \(MSE\). An externally studentized, or deleted, residual uses an error variance estimate from a refit that omits the observation. The latter can be especially useful because an extreme observation can inflate the full-data \(MSE\), making its own internally studentized residual look less unusual.

A studentized residual is a diagnostic, not a verdict. Values with large absolute magnitude merit investigation, but there is no universal cutoff that makes an observation an outlier in every setting. When many observations are checked, some large residuals can occur by chance. Consider the plot, the data-collection process, units, and possible recording errors; do not use a numerical flag as an automatic deletion rule.

How Deletion Changes a Group Mean

The effect of deleting an observation on its own group's mean has an exact, simple formula. Write \(\bar{x}_i\) for the original mean and \(\bar{x}_{i(-j)}\) for the mean after deleting \(y_{ij}\). Provided \(n_i\ge 2\), the deleted mean is the average of the other \(n_i-1\) observations:

Subtracting this from the original mean gives the change. Since \(e_{ij}=y_{ij}-\bar{x}_i\), the size and direction of that change are determined by the residual and group size.

The deleted mean moves away from the observation: if \(e_{ij}\) is positive, removing the observation lowers the group mean. For the same residual, a smaller group has a larger change in its mean. This explains why residual size and influence should not be treated as interchangeable: the mean's sensitivity also depends on \(n_i\).

Refitting the ANOVA After a Flag

A useful influence check is to calculate the ANOVA using all valid observations, then refit it after temporarily omitting a flagged observation. Compare group means, \(SSE\), \(MSE\), degrees of freedom, and the F statistic. If the analysis includes post hoc comparisons, those too may change and must be recalculated under the chosen procedure. A changed result is evidence that the observation matters to the analysis; it is not, on its own, a reason to discard it.

The deletion formula for \(SSE\) can also be derived directly. Within group \(i\), the residuals sum to zero. After deleting \(y_{ij}\), each remaining residual from the new group mean is \(e_{i\ell}+e_{ij}/(n_i-1)\). Squaring and summing over the other observations, using \(\sum_{\ell\ne j}e_{i\ell}=-e_{ij}\), gives

The other groups' sums of squares do not change. Thus, for a deletion that leaves every group represented, \(SSE_{(-ij)}=SSE-\frac{n_i}{n_i-1}e_{ij}^2\). The refitted model has \(N-1\) observations and still estimates \(k\) group means, so its error degrees of freedom are \(N-k-1\). This derivation, and the deleted-mean formulas above, require \(n_i\ge2\). If the observation is the only one in its group, deleting it removes that group, and these formulas and degrees of freedom do not apply.

Interpreting and Reporting Flags

A careful diagnostic workflow separates three questions: Is the observation unusual relative to its fitted group mean? Does removing it materially change the fitted means or inferential results? Is there a defensible, documented reason to correct or exclude it? Possible reasons include a verified recording or measurement error, but an inconvenient result or a large residual is not sufficient justification. If a value is valid, the primary analysis should ordinarily retain it, with a clearly described sensitivity analysis when its influence is important.

When reporting a sensitivity analysis, state the reason the observation was flagged, how it was handled, and how the conclusions differed with and without it. Do not present only the version that supports a preferred conclusion. If an observation is removed because a value was verified to be erroneous, describe the correction or exclusion and report the analysis based on the corrected data. If it remains a valid observation, make clear which analysis is primary and which is the sensitivity check.