Why Multiple Unusual Cases Need Special Attention
A regression diagnostic evaluates each observation relative to a fitted model. That creates a difficulty when several observations are unusual together: the fitted model itself may already have been pulled toward them. A case can then look less unusual than it would against a fit based on the remaining data. Conversely, observations that are ordinary under the underlying pattern can look unusual because a group has distorted the fitted line or inflated the residual variation.
These effects are called masking and swamping. They are reasons to compare fits and diagnostics rather than treat one residual or influence measure as a final verdict. The earlier material on studentized residuals, leverage, influence measures, and case-deletion fitting provides useful tools for these comparisons.
For context, consider the usual linear regression model \(Y=X\beta+\varepsilon\), where the design matrix \(X\) has full column rank and the errors have mean zero, common variance \(\sigma^2\), and are uncorrelated. Normal errors are additionally assumed when exact normal-theory inference is required. The least-squares estimate is \(\hat{\beta}=(X^{\mathsf T}X)^{-1}X^{\mathsf T}y\), and the residual for case \(i\) is \(e_i=y_i-x_i^{\mathsf T}\hat{\beta}\). These model assumptions support the usual regression calculations; they do not guarantee that every observed case follows the model. In particular, the deletion identities below require full rank after the case is removed. They do not apply if the reduced design matrix is singular.
Compare the Full Fit With Deletion Fits
A useful first comparison is between a case’s residual in the full fit and its residual when that case is excluded from the estimation. Write \(\hat{\beta}_{(i)}\) for the estimate after deleting case \(i\), and define its case-deletion residual as \(e_{(i)}=y_i-x_i^{\mathsf T}\hat{\beta}_{(i)}\). For a group \(G\), refit after deleting every case in \(G\), then examine both the change in the fitted model and the residuals of the deleted cases relative to that model.
Here \(h_{ii}\) is the diagonal leverage for case \(i\). This relation is for a single deleted case; it does not provide a group-deletion residual by substituting a group leverage into the same formula. For groups, refit the model or use a valid group-deletion calculation.
The identity explains why an observation’s full-fit residual can be small when its leverage is high: \(e_i=(1-h_{ii})e_{(i)}\). A small \(1-h_{ii}\) can shrink the full-fit residual even when the deleted-case residual is much larger. This is one route to masking. It is not, by itself, proof of an outlier; the residual must still be interpreted with the model, the scale estimate, and the data context in view.
Worked Example: A Pair That Pulls the Fitted Line
A hypothetical operations team models a centered process score \(y\) against a centered operating index \(x\). Five routine records are \((-2,-2),(-1,-1),(0,0),(1,1),(2,2)\). Two additional records are \((10,25)\) and \((11,27)\). Compare the full least-squares line with the line fitted to the five routine records. What diagnostic pattern does the comparison suggest?
Method. Fit the full model \(y=\beta_0+\beta_1x+\varepsilon\), then delete the two high-index records as a group and refit. The ordinary least-squares model is being used descriptively; this comparison alone does not establish why the records differ.
For all seven records, \(\sum x=21\), \(\sum y=52\), \(\sum x^2=231\), and \(\sum xy=557\). Therefore \(\bar{x}=21/7=3\), \(\bar{y}=52/7=7.428571\), and
The fitted slope is \(401/168=2.386905\), and the intercept is \(7.428571-(2.386905)(3)=0.267857\). Thus the full-fit line is \(\hat{y}=0.267857+2.386905x\). Its residuals at the two added records are \(25-(0.267857+2.386905(10))=0.863095\) and \(27-(0.267857+2.386905(11))=0.476190\). Both are modest compared with the departure of either record from the routine-data line.
Deleting the two added records leaves the five points exactly on \(\hat{y}=x\). Relative to this group-deleted fit, the two excluded records have residuals \(25-10=15\) and \(27-11=16\). The group has pulled the full-fit slope away from 1 and made its own residuals much smaller than their departures from the routine pattern. That is evidence of possible mutual masking. The full-fit residuals for routine cases are also no longer zero: at \(x=2\), for example, the residual is \(2-(0.267857+2.386905(2))=-3.041667\). Their residuals shrink to zero under the routine-only fit, illustrating how a contaminated fit can make routine cases look less consistent with the assumed pattern. No automatic deletion follows; the records and model assumptions require investigation.
Check. Substituting \(x=10\) and \(x=11\) in the full line gives fitted values 24.136905 and 26.523810, respectively; subtracting these from 25 and 27 reproduces the residuals above. The common mistake is to judge the pair only by their full-fit residuals and overlook how much the fit changes when they are removed together.
Separating Masking From Swamping
A disciplined comparison asks what changes under several fits. First identify cases that may be unusual using the full-fit residuals, leverage, and influence diagnostics. Then compare individual deletion fits, because one case may be concealed by another. Finally, if the data suggest a cluster, delete the suspected group together and examine the coefficient changes, residual scale, and the deleted cases’ departures from the reduced fit. For a group-deleted model, predictions for the deleted cases can be compared with their observed responses, but they are not ordinary in-sample residuals from that reduced fit.
Masking and swamping can occur in the same data set. A cluster may pull the fitted line toward itself, shrinking its own residuals, while making some routine observations appear to depart from the line. A change in residual size after deletion is informative, but so is a change in the estimated residual standard deviation: a group may inflate the scale and make other observations seem less striking when standardized.
Worked Example: Interpreting Leave-One-Out Evidence for Two Cases
A hypothetical regression of a laboratory response on two predictors includes cases A and B, which have been flagged for review. Diagnostic output from a full fit reports \(e_A=0.64\), \(h_{AA}=0.92\), \(e_B=0.72\), and \(h_{BB}=0.91\). A separate fit using unaffected reference records predicts responses 12 units below the observed values for both A and B. Calculate each leave-one-out residual and interpret the comparisons.
Method. Apply the single-case deletion identity to each case, then compare those results with the reference-fit departures. For A,
For B,
The full-fit residuals, 0.64 and 0.72, are much smaller than their leave-one-out residuals of 8. The reference fit’s departure is 12 for each. The three comparisons suggest that each observation is partly accommodated by the fit that includes it, and that neither is fully explained by the other’s presence. The large leverage values account for why the full-fit residuals are shrunk. The data therefore warrant examining A and B jointly as well as individually; the supplied summaries do not establish that they are invalid.
Check. \(0.08(8)=0.64\) and \(0.09(8)=0.72\), matching the reported full-fit residuals. A common mistake is to treat the full-fit residual as the case’s departure from a model estimated without that case.
Worked Example: Recognizing Possible Swamping After Group Deletion
A hypothetical model for claim-processing time has a full-fit residual standard error \(s=2\). For an ordinary record C, the full-fit residual is \(e_C=-2.4\) and its leverage is \(h_{CC}=0.40\). After a separate group of flagged records is removed and the model is refitted, C’s residual is \(-0.3\), its leverage in that fit is \(0.20\), and the reduced fit’s residual standard error is \(1.5\). Calculate C’s internally standardized residual in each fit and interpret the change.
Method. For a fitted least-squares model with residual standard error \(s\), the internally standardized residual is \(r_i=e_i/[s\sqrt{1-h_{ii}}]\). This scaling accounts for the fitted variance of a residual under the stated regression assumptions. It is not a universal cutoff or a guarantee that the model is correct.
In the full fit, the denominator is \(2\sqrt{1-0.40}=2\sqrt{0.60}=1.54919\), so \(r_C=-2.4/1.54919=-1.549\). In the group-deleted fit, the denominator is \(1.5\sqrt{1-0.20}=1.5\sqrt{0.80}=1.34164\), so \(r_C=-0.3/1.34164=-0.224\). Rounded to three decimals, the standardized residual changes from \(-1.549\) to \(-0.224\). C appears much less unusual after the flagged group is excluded, which is consistent with possible swamping in the full fit. This conclusion is diagnostic: other changes in the model or data could also affect the residual.
Check. The calculations use each fit’s own residual, leverage, and residual standard error. Using the full-fit standard error with the reduced-fit residual would mix two models and would not give the stated standardized residual. The common mistake is to attribute every improvement after deletion to swamping without checking that the cases removed were themselves reasonably identified.
Using the Comparison Responsibly
Case-deletion comparisons are most useful when interpreted as a pattern of evidence. A strong change in coefficients indicates that the fitted relationship depends on the deleted cases. A large difference between full-fit and deleted-case residuals can indicate masking. A routine case whose standardized residual becomes much smaller after group deletion may have been swamped. None of these statements is equivalent to “delete the case”: unusual observations may represent data errors, a different population, a missing predictor, or genuine variation that the model does not capture.
The full-rank condition matters in practice. Removing a group can leave too few observations or remove the only information for estimating one or more coefficients. In that event, the reduced model is not estimable in the same form, so its coefficient and residual comparisons cannot be interpreted as ordinary case-deletion results. Also, when many plausible groups are tried, the most dramatic comparison may reflect the search process rather than a prespecified diagnostic question.
Review residuals, leverage, and influence together; do not label a case from one statistic alone.
Compare leave-one-out residuals and coefficient changes to see whether a case is concealed by another.
Refit after deleting the group together, provided the reduced design remains estimable, and compare the model and residual scale.
Check data quality, context, and model specification; a diagnostic flag is a reason to learn more, not an instruction to delete.
Check Your Understanding
For each problem, distinguish the arithmetic conclusion from the diagnostic interpretation.
- A full-fit residual is \(0.45\) and leverage is \(0.85\). Calculate the leave-one-out residual using the single-case deletion identity. (Answer: \(0.45/(1-0.85)=3.00\).)
- Two cases have full-fit residuals 0.5 and 0.6, while a fit excluding both predicts each response 9 units below its observed value. What pattern is suggested, and what can you not conclude from this comparison alone? (Answer: Possible mutual masking; the comparison alone does not prove either case is invalid or justify deletion.)
- An ordinary case has full-fit residual \(-1.8\), leverage \(0.19\), and residual standard error \(1.5\). Calculate its internally standardized residual to three decimals. (Answer: \(-1.8/[1.5\sqrt{0.81}]=-1.333\).)
- For a case, \(e_i=0.24\) and \(h_{ii}=0.94\). Calculate its leave-one-out residual and explain why the full-fit residual may be misleading. (Answer: \(0.24/0.06=4\); high leverage makes the full-fit residual much smaller than the deleted-case residual.)
- After a suspected group is deleted, an ordinary case’s standardized residual changes from 2.1 to 0.4. What diagnostic effect is consistent with this change, and name one additional check before drawing a conclusion. (Answer: Possible swamping; check that the deleted group is supported by case-level evidence and that the reduced model remains estimable.)