Case-Deletion Effects on ANOVA Comparisons

Why Examine an Effect After Deleting One Observation?

An ANOVA estimate summarizes information from a collection of observations. A treatment mean, for instance, depends on every observation in that treatment group. It is useful to ask how much that estimate would change if one observation were omitted. The same question applies to a treatment contrast, such as the difference between two group means, and to fitted effects in a more general ANOVA linear model.

This is a sensitivity analysis: the aim is to describe how dependent an estimate is on a particular case, not to decide that the case is invalid. A large change flags an estimate for careful inspection and contextual judgment. Case deletion is not a substitute for checking the data, and a case should not be removed merely because its removal makes a result more appealing.

Leave-One-Out Estimates and DFBETAS

Let \(y_i\) be observation \(i\), belonging to group \(g\), with group size \(n_g\) and sample mean \(\bar{x}_g\). Write \(\bar{x}_{g(-i)}\) for that group’s mean after observation \(i\) is removed. Other groups’ means do not change in a one-way ANOVA when only this observation is deleted.

For a group mean, the change has a particularly simple form: \(\bar{x}_g-\bar{x}_{g(-i)}=(y_i-\bar{x}_g)/(n_g-1)\). Thus a case above its group mean pulls the full-data mean upward relative to the leave-one-out mean; a case below the mean pulls it downward. The size of the shift also depends on group size: the same deviation generally matters more in a smaller group.

For a contrast, such as \(\mu_A-\mu_B\), calculate the contrast from the full-data group means and again from the leave-one-out group means. A deleted observation in group A can change this contrast; an observation in a different group will affect it only if that group appears in the contrast. A contrast can shift even when no single group mean appears especially sensitive, because its component means may move in different directions.

A General Linear-Model Description

The one-way ANOVA model is a special case of the fixed-effects linear model. Let \(y\) be the vector of \(N\) observed responses, \(X\) the design matrix, and \(\boldsymbol{\beta}\) the coefficient vector. The model is \(y=X\boldsymbol{\beta}+\boldsymbol{\varepsilon}\). Assume that \(X\) has full column rank and that the errors are independent with mean zero and common variance \(\sigma^2\). Normal errors are additionally needed for exact normal-theory inference, but not for the least-squares deletion identity below. In an example, the design and data collection should support independence and a common-variance model; small data sets alone cannot establish normality or equal variances.

Write \(\hat{\boldsymbol{\beta}}\) for the least-squares estimate, \(e_i=y_i-\hat{y}_i\) for residual \(i\), \(h_{ii}\) for its hat value, and \(s_{(-i)}\) for the residual standard deviation after deleting case \(i\). For coefficient \(j\), let \(c_{jj,(-i)}\) be diagonal entry \(j\) of \((X_{(-i)}^\mathsf{T}X_{(-i)})^{-1}\), where \(X_{(-i)}\) is the design matrix with row \(i\) removed. The DFBETAS convention used here is:

The denominator is the deleted-model standard error for coefficient \(j\). For a linear effect \(L=\mathbf{c}^{\mathsf{T}}\boldsymbol{\beta}\), use the corresponding estimate difference and the deleted-model standard error \(s_{(-i)}\sqrt{\mathbf{c}^{\mathsf{T}}(X_{(-i)}^\mathsf{T}X_{(-i)})^{-1}\mathbf{c}}\). DFBETAS is therefore a signed, standardized change: the sign records the direction of full estimate minus deleted estimate. Other software may use the opposite sign convention, so check the convention before comparing reported signs. Its magnitude is the focus when assessing the size of a shift.

Why the Deletion Formula Works

For a full-rank least-squares model, deleting one case changes the coefficient vector according to a compact identity. It connects the change in fitted effects to the case’s residual and leverage. The identity assumes that deleting the case leaves the reduced design matrix full rank.

To verify the identity, the full least-squares normal equations give \(X^\mathsf{T}e=0\). After deleting case \(i\), the normal-equation right-hand side is \(X^\mathsf{T}y-\mathbf{x}_i y_i\), and the cross-product matrix is \(X^\mathsf{T}X-\mathbf{x}_i\mathbf{x}_i^\mathsf{T}\). The full fit satisfies \(X^\mathsf{T}y=X^\mathsf{T}X\hat{\boldsymbol{\beta}}\), while \(y_i=\mathbf{x}_i^\mathsf{T}\hat{\boldsymbol{\beta}}+e_i\). Substituting these expressions into the reduced normal equations gives \((X^\mathsf{T}X-\mathbf{x}_i\mathbf{x}_i^\mathsf{T})\hat{\boldsymbol{\beta}}_{(-i)} =X^\mathsf{T}X\hat{\boldsymbol{\beta}}-\mathbf{x}_i(\mathbf{x}_i^\mathsf{T}\hat{\boldsymbol{\beta}}+e_i)\). Rearranging yields \((X^\mathsf{T}X-\mathbf{x}_i\mathbf{x}_i^\mathsf{T})(\hat{\boldsymbol{\beta}}-\hat{\boldsymbol{\beta}}_{(-i)}) =\mathbf{x}_i e_i\). The solution is the stated identity: multiplying its right-hand side by the reduced cross-product matrix gives \((X^\mathsf{T}X-\mathbf{x}_i\mathbf{x}_i^\mathsf{T})(X^\mathsf{T}X)^{-1}\mathbf{x}_i e_i/(1-h_{ii})=\mathbf{x}_i e_i\), since \(h_{ii}=\mathbf{x}_i^\mathsf{T}(X^\mathsf{T}X)^{-1}\mathbf{x}_i\). This justifies the formula without requiring the full model to be refit from scratch for each case.

Interpreting the Diagnostic Carefully

A large absolute DFBETAS value means that deleting the case changes the specified coefficient by many deleted-model standard errors. It does not, by itself, show that the observation is wrong, that the model is inappropriate, or that a scientific conclusion must change. Conversely, a small DFBETAS for one coefficient does not imply that the case has little effect on every other coefficient or contrast. Influence is tied to the effect being examined.

A common rough screening guide in regression is to flag values whose absolute DFBETAS exceed \(2/\sqrt{N}\). This is a heuristic, not a universal decision boundary: its usefulness depends on the design, the number of effects examined, and the purpose of the analysis. Report the estimate and its leave-one-out counterpart when possible, because the raw difference preserves the units and practical meaning that a standardized value can obscure.

Keep the question focused: which treatment mean, contrast, or fitted coefficient is sensitive, by how much does it change, and in what direction? These are statements about the stability of estimates. Whether deletion changes an ANOVA test or the resulting conclusion is a separate question; an estimate can shift without altering a test decision, and a modest shift can matter near a decision boundary.