Why Groups of Cases Complicate Influence Screening
Influence screening often examines observations one at a time: compare each case’s residual, leverage, or change in the fitted model with a reference rule. But cases do not always act independently. A group of unusual observations can collectively pull the regression fit toward itself, making each member seem less unusual than it would against a fit based on the rest of the data. Conversely, that altered fit can make ordinary observations look unusual.
These two effects are called masking and swamping. They are reasons to treat diagnostics as evidence for review rather than automatic decisions. In particular, a modest diagnostic for each case individually does not guarantee that the dataset contains no influential group.
Model and Diagnostic Summaries
For a dataset with \(n\) cases, consider an ordinary least squares regression with \(p\) fitted coefficients, including the intercept. Write the model as \(y=X\beta+\varepsilon\), where \(X\) is the design matrix and the columns of \(X\) are linearly independent. The fitted values are \(\hat{y}=X\hat{\beta}\), and the residuals are \(e_i=y_i-\hat{y}_i\). The hat matrix is \(H=X(X^\mathsf{T}X)^{-1}X^\mathsf{T}\); its diagonal element \(h_{ii}\) is the leverage of case \(i\).
For the usual model-based interpretation of these diagnostics, assume the regression mean is correctly specified and the errors have mean zero, constant variance, and are independent across cases. Normal errors are additionally needed for exact small-sample t-distribution results for studentized residuals. The calculations below describe the diagnostics themselves; a striking value is not, by itself, proof that the model assumptions hold or fail.
The residual standard error is \(s=\sqrt{\mathrm{SSE}/(n-p)}\), where \(\mathrm{SSE}=\sum_i e_i^2\). An internally studentized residual scales a residual using this full-fit estimate and its leverage. Cook’s distance summarizes how much the fitted values would change when a case is deleted. These summaries capture different features: residual size, predictor-space position, and effect on the fitted model.
Worked Example: A Pair of Endpoints Masks Its Joint Pattern
Worked Example: A Pair of Endpoints Masks Its Joint Pattern
Suppose five hypothetical cases in a simple regression have predictor values \(x=(-2,-1,0,1,2)\) and responses \(y=(-10,0,0,0,10)\). The three interior cases follow the horizontal pattern \(y=0\); the two endpoints lie on a different trend. Find the full-data regression and compare its summaries across the cases.
For simple regression with an intercept, \(p=2\). The predictor mean and response mean are both zero. Therefore the fitted intercept is zero, and the slope is
Thus the fitted line is \(\hat{y}=4x\). The fitted values are \((-8,-4,0,4,8)\), so the residuals are \((-2,4,0,-4,2)\). Notice that each endpoint’s residual is only 2 response units, although each endpoint differs by 10 units from the horizontal pattern of the three interior cases.
For simple regression with an intercept, \(h_{ii}=1/n+(x_i-\bar{x})^2/\sum_j(x_j-\bar{x})^2\). Here the leverages are \(0.60,0.30,0.20,0.30,0.60\). Also, \(\mathrm{SSE}=4+16+0+16+4=40\), so \(s^2=40/(5-2)=13.333\) and \(s=3.651\).
At either endpoint, the internally studentized residual has magnitude \(2/[3.651\sqrt{1-0.60}]=0.8660\). At either case with \(x=-1\) or \(x=1\), its magnitude is \(4/[3.651\sqrt{1-0.30}]=1.309\). The endpoints have greater leverage, but the interior cases have larger internally studentized residuals.
For either endpoint, Cook’s distance is \(4/[2(13.333)]\times[0.60/(0.40)^2]=0.5625\). For either case with \(x=-1\) or \(x=1\), it is \(16/[2(13.333)]\times[0.30/(0.70)^2]=0.3673\); at \(x=0\), it is zero because that case’s residual is zero. The endpoint cases have the larger Cook’s distances in this dataset, while the interior cases have the larger internally studentized residuals. The summaries are not interchangeable.
The group effect becomes clear by fitting only the three interior cases: their fitted line is \(y=0\). Relative to that line, each endpoint differs by 10 response units; relative to the full-data line, each endpoint’s residual is only 2. The endpoint pair has pulled the full-data slope toward itself, reducing its own residuals. That is masking. The interior cases, in turn, have residuals of magnitude 4 under the full-data line, even though they all lie exactly on their own horizontal pattern.
Worked Example: A Shifted Fit Can Swamp Ordinary Cases
Worked Example: A Shifted Fit Can Swamp Ordinary Cases
Consider a second hypothetical dataset with the same predictor values, \(x=(-2,-1,0,1,2)\), but responses \(y=(10,0,0,0,10)\). The endpoints form a group with responses above the interior pattern. Find the fitted line and compare the residual sizes.
By symmetry, the predictor-response cross-product sum is zero, so the fitted slope is zero. The mean response is \((10+10)/5=4\), giving the fitted line \(\hat{y}=4\). The fitted values are \((4,4,4,4,4)\), and the residuals are \((6,-4,-4,-4,6)\).
The endpoints’ observed responses are 10 units above the interior level of zero, but their full-fit residuals are 6 because the endpoint group has shifted the fitted intercept upward. The three interior cases now each have residual \(-4\), although they share the same response pattern. This illustrates how a group’s distortion of the fit can make ordinary cases appear unusual: a screen based on residual magnitude might flag interior cases, depending on its chosen rule.
The leverage values are still \(0.60,0.30,0.20,0.30,0.60\). Here \(\mathrm{SSE}=2(6^2)+3(4^2)=120\), so \(s^2=120/3=40\). An endpoint Cook’s distance is \(36/(2\cdot40)\times[0.60/(0.40)^2]=1.6875\). At either \(x=-1\) or \(x=1\), it is \(16/(2\cdot40)\times[0.30/(0.70)^2]=0.1224\). In this example, Cook’s distance makes the endpoint group’s influence more evident than the interior cases’ influence. No one number, however, establishes which cases are valid or what model should replace the current one.
Why Deleting a Group Can Tell a Different Story
A useful way to understand masking is to compare the full fit with a fit that omits a group, while remembering that group deletion is a diagnostic comparison rather than an instruction to discard cases. The following calculation uses the first example.
Worked Example: Compare Single-Case and Group Deletion
In the first dataset, the full-data line is \(\hat{y}=4x\), and the endpoint residuals are \(-2\) and \(2\). Delete both endpoints together. The remaining three cases have \(x=(-1,0,1)\) and \(y=(0,0,0)\), so their least-squares line is \(\hat{y}=0\). The omitted endpoints then differ from this line by \(-10\) and \(10\) response units.
Now delete only the case at \(x=-2\). The four remaining points are \((-1,0),(0,0),(1,0),(2,10)\). Their means are \(\bar{x}=0.5\) and \(\bar{y}=2.5\). Their centered cross-product sum is \(20-4(0.5)(2.5)=15\), and their centered predictor sum of squares is \(6-4(0.5)^2=5\). Thus the deleted-case fit has slope \(15/5=3\) and intercept \(2.5-3(0.5)=1\). At \(x=-2\), this line predicts \(-5\), leaving a deleted-fit residual of \(-10-(-5)=-5\), larger in magnitude than the full-fit residual of \(-2\).
The group comparison reveals a stronger departure from the interior pattern than either endpoint’s residual under the full fit. The single-deletion comparison also changes the endpoint’s prediction. These contrasts explain why one-at-a-time summaries can understate a coordinated pattern; they do not mean that a group-deletion fit is automatically the right model.
A Short Derivation: How a Group Pulls the Line
The slope calculation in the first example can be understood directly. For the symmetric predictor values, \(\bar{x}=0\), and the two endpoint products \(x_i y_i\) are both positive: \((-2)(-10)=20\) and \((2)(10)=20\). The interior cases contribute zero to \(\sum_i x_i y_i\). Consequently, the endpoints contribute 40 to the slope numerator while the total predictor sum of squares is 10, producing slope 4. The three interior cases alone would give numerator zero and slope zero. Thus their joint contribution changes the fitted slope, and the resulting line moves toward both endpoints. This is the mechanism behind masking in this example.
How to Read a Group Pattern
A practical interpretation compares several kinds of evidence rather than counting flags in isolation. Residual size asks how far a response lies from the fitted value. Leverage describes how unusual a case’s predictor values are relative to the others. Cook’s distance summarizes the case’s effect on fitted values. Deleted-fit comparisons can show whether the story changes when a case or a plausible group is removed. Agreement among summaries can strengthen the reason to investigate; disagreement is itself informative about how a case affects the fit.
When several cases appear related in predictor space or share a pattern among their responses, inspect them together as well as individually. Compare the fitted model and residual pattern with and without the group, and check whether the conclusion depends on a few cases. A group-based comparison should be motivated by the data or subject-matter context, not selected solely because it makes a preferred result appear.
Common Mistake
Wrong reasoning: “No individual case has an especially large residual, so there cannot be an unusual group.” This overlooks the fact that the group may have changed the line against which its own residuals are measured.
Correction: Examine the joint pattern and compare relevant summaries or fits, including a justified group-deletion comparison when appropriate. Also avoid the opposite mistake of treating every flagged ordinary case as genuinely unusual: a group can shift the fit and create apparent departures among cases that otherwise follow a coherent pattern. A diagnostic flag is a prompt to check the data and model, not a verdict.
Check Your Understanding
Use the examples and definitions above to answer these questions.
- In your own words, distinguish masking from swamping.
- In the first worked example, why are the endpoints’ full-fit residuals smaller than their differences from the line fitted to the interior cases?
- Why can high leverage and a small residual occur together?
- In the second example, what effect does the endpoint group have on the fitted intercept, and how do the interior residuals change as a result?
- Why is it useful to compare more than one influence diagnostic?