Masking and Swamping in Outlier Detection

Why Several Unusual Cases Can Mislead a Screen

A residual is the difference between an observed response and the response predicted by a fitted regression model. It is tempting to treat a large residual as a direct measure of how unusual an observation is. But the fitted line is calculated from the observations themselves. If several cases are unusual in a similar way, they can pull the line toward themselves and make their own residuals smaller. At the same time, the line may shift away from ordinary cases, making those cases look unusual instead.

These two effects are called masking and swamping. They matter because a residual-based screen is not independent of the data being screened: the candidate observations help determine the model against which their residuals are measured.

Definition: Masking occurs when one or more unusual observations make their own departures from a fitted model look smaller, so that a residual-based screen fails to flag them. Swamping occurs when unusual observations distort the fitted model enough that ordinary observations appear to have unusually large residuals and are flagged.

The Model and the Screening Idea

For a simple linear regression, the fitted line has the form \(\hat{y}=b_0+b_1x\). Here \(b_0\) is the fitted intercept, \(b_1\) is the fitted slope, and the residual for observation \(i\) is \(e_i=y_i-\hat{y}_i\). Least squares chooses \(b_0\) and \(b_1\) to minimize the sum of squared residuals.

A residual screen compares residuals with a reference rule. In the examples below, the rule is simply to flag any case with \(|e_i|>5\) response units. This is a deliberately fixed teaching rule, not a general recommendation or a formal significance test. A useful screen also depends on a suitable model for the conditional mean of \(Y\) given \(X\); otherwise, a pattern the model cannot represent may look like outlying responses. Independence and a reasonably stable residual scale also matter when residuals are interpreted as evidence about unusual cases. The numerical examples use exact patterns to isolate the mechanics; they do not require a normal-error assumption, and they make no probability claims.

The least-squares slope is \(b_1=S_{xy}/S_{xx}\), where \(S_{xy}=\sum (x_i-\bar{x})(y_i-\bar{y})\) and \(S_{xx}=\sum (x_i-\bar{x})^2\). Adding cases can change both quantities. Cases with unusual predictor values can have especially substantial effects because their squared distances contribute to \(S_{xx}\), while their positions relative to the response trend contribute to \(S_{xy}\). Thus the fitted line—and the residuals used to screen cases—can change together.

Worked Examples of Masking and Swamping

Worked Example: Added Cases Pull the Line Toward Themselves

Suppose five ordinary cases follow \(y=2x\), with predictor values \(-2,-1,0,1,2\) and responses \(-4,-2,0,2,4\). Now consider adding a case at \(x=4\) with \(y=21\), and then a second case at \(x=5\) with \(y=22\). These added responses are well above the values \(8\) and \(10\) predicted by the ordinary pattern. Use the illustrative rule \(|e_i|>5\).

First, add only the case \((4,21)\). For all six cases, \(\sum x_i=4\), \(\sum y_i=21\), \(\sum x_i^2=26\), and \(\sum x_iy_i=104\). Therefore,

S_{xx}=26-\frac{4^2}{6}=\frac{70}{3}, \qquad S_{xy}=104-\frac{(4)(21)}{6}=90.

The fitted slope is \(b_1=90/(70/3)=27/7\approx3.857\), and the intercept is \(b_0=21/6-(27/7)(4/6)=13/14\approx0.929\). At \(x=4\), the fitted response is \(229/14\approx16.357\), so the added case has residual \(21-229/14=65/14\approx4.643\). It is not flagged by the rule.

Now add the second case \((5,22)\). For all seven cases, \(\sum x_i=9\), \(\sum y_i=43\), \(\sum x_i^2=51\), and \(\sum x_iy_i=214\). Thus,

S_{xx}=51-\frac{9^2}{7}=\frac{276}{7}, \qquad S_{xy}=214-\frac{(9)(43)}{7}=\frac{1111}{7}.

The new slope is \(b_1=1111/276\approx4.025\), and the intercept is \(b_0=43/7-(1111/276)(9/7)=623/644\approx0.967\). The fitted responses at \(x=4\) and \(x=5\) are about \(17.069\) and \(21.094\). Their residuals are therefore \(21-17.069=3.931\) and \(22-21.094=0.906\), respectively. Neither is flagged by the rule, even though both added responses depart substantially from the ordinary pattern. The second case helped pull the line toward the pair, reducing the residual magnitude at \((4,21)\).

For comparison, the ordinary case at \(x=2\) has fitted response about \(9.018\) and residual \(4-9.018\approx-5.018\), so it is flagged by the illustrative rule. The added case's residual decreases from about \(4.643\) when fitted alone to about \(3.931\) when both added cases are included, but it is below the cutoff in both fits. Thus, this example illustrates a reduction in residual magnitude, not a screen-based masking change. It does not establish that a particular case should be removed.

Worked Example: Unusual Cases Swamp Ordinary Cases

Use the same five ordinary cases on \(y=2x\), but add \((4,-24)\) and \((5,-30)\). The added responses are far below the ordinary pattern's predictions, \(8\) and \(10\). Again, use \(|e_i|>5\) as an illustrative flagging rule.

Across all seven cases, \(\sum x_i=9\), \(\sum y_i=-54\), \(\sum x_i^2=51\), and \(\sum x_iy_i=-226\). The centered sums are

S_{xx}=51-\frac{9^2}{7}=\frac{276}{7}, \qquad S_{xy}=-226-\frac{(9)(-54)}{7}=-\frac{1096}{7}.

Hence \(b_1=(-1096/7)/(276/7)=-274/69\approx-3.971\). The intercept is \(b_0=-54/7-(-274/69)(9/7)=-60/23\approx-2.609\). The fitted line is approximately \(\hat{y}=-2.609-3.971x\), a striking change from the ordinary pattern's slope of \(2\).

At the ordinary case \(x=2,y=4\), the fitted response is \(-2.609-3.971(2)\approx-10.551\), giving residual \(4-(-10.551)\approx14.551\). That ordinary case is flagged. The ordinary case at \(x=1,y=2\) has residual about \(8.580\), also flagged. Meanwhile, the two added cases have residuals of approximately \(-5.507\) and \(-7.536\), so they too are flagged by this cutoff. The false flags among ordinary cases are an example of swamping: the unusual cases have pulled the line away from much of the ordinary pattern.

If only the five ordinary cases were fitted, the least-squares line would be exactly \(\hat{y}=2x\), and all five residuals would be zero. That comparison clarifies what is meant by “ordinary” in this constructed example. In actual data, the true pattern is not known so neatly, and residual flags alone cannot prove which cases are ordinary.

Worked Example: An Unusual Predictor Is Not Automatically an Unusual Response

Consider four hypothetical observations: \((0,1)\), \((1,3)\), \((2,5)\), and \((10,21)\). The last predictor value is much farther from the others, but all four responses follow \(y=1+2x\). Fit a least-squares line and check the residuals.

Here \(n=4\), \(\bar{x}=3.25\), and \(\bar{y}=7.5\). Also, \(\sum x_i^2=105\) and \(\sum x_iy_i=223\), giving

S_{xx}=105-4(3.25)^2=62.75, \qquad S_{xy}=223-4(3.25)(7.5)=125.5.

The slope is \(125.5/62.75=2\), and the intercept is \(7.5-2(3.25)=1\). Each observed response equals its fitted response, so all four residuals are zero. A screen based only on response residuals does not flag the case at \(x=10\), despite its unusual predictor value. Whether a predictor value is unusual and whether a response departs from the fitted relationship are different questions.

A Short Justification: Why the Flags Can Change

The mechanism can be seen directly from the residual definition. For any observation, \(e_i=y_i-b_0-b_1x_i\). If added cases shift \(b_0\) or \(b_1\) toward their responses, their fitted values move closer to their observed values and their residual magnitudes can shrink. In the first example, the extra case changed the slope from about \(3.857\) to \(4.025\), reducing the residual at \((4,21)\) from about \(4.643\) to \(3.931\).

The same coefficient change affects every other case. In the swamping example, the added cases changed the slope from the ordinary-pattern value \(2\) to about \(-3.971\). The fitted value at \(x=2\) consequently moved from \(4\) to about \(-10.551\), turning a zero residual into a residual of about \(14.551\). The calculations show that a residual is conditional on the fitted model; it is not a fixed measure attached to an observation regardless of the other data.

How to Interpret a Residual Screen

Masking and swamping are warnings about the reference line, not proof that any particular case is incorrect. Several questions should be considered together:

Is the model suitable? A curved or otherwise misspecified mean relationship can create residual patterns that do not represent isolated outliers.
Are cases unusual in the predictor, the response, or both? A distant predictor value can be consistent with the fitted response pattern, as the third example shows.
Does the conclusion depend on the other unusual cases? A flag that appears or disappears as the candidate group changes deserves careful investigation. This is a diagnostic comparison, not a mechanical deletion rule.
Are the cutoff and residual scale appropriate? The same numerical cutoff is not meaningful in every response's units or at every residual scale. Standardized diagnostics use scale information, but they too require suitable assumptions and careful interpretation.

Residuals from one least-squares fit are linked by the fitting process, so one should not interpret a collection of flags as independent pieces of evidence. In particular, flagging an ordinary case does not mean that its recorded value is wrong; it may be unusual only relative to a line distorted by other cases. Nor does failing to flag a case establish that it follows the intended pattern.

Common Mistake

Wrong reasoning: “The residual is below the cutoff, so this case cannot be an outlier.” This ignores that other unusual observations may have pulled the fitted line toward it. The opposite error is to conclude that every case above a cutoff is erroneous.

Correction: Treat a residual flag as a prompt to investigate the observation and the model together. Check the data and context, assess whether the model represents the relationship adequately, and consider whether a group of cases may be influencing the reference line. Do not equate a statistical flag with a data error or an automatic deletion decision.

Key takeaway: Masking can hide unusual responses by pulling the fitted line toward them; swamping can make ordinary responses appear unusual by pulling the line away from them. Residual flags are model-dependent evidence that calls for investigation, not verdicts about individual records.

Check Your Understanding

Use the definitions and examples to answer these questions.

In your own words, distinguish masking from swamping.
Why can two unusual cases have smaller residuals together than one unusual case has when fitted alone?
In the swamping example, what happened to the fitted slope, and how did that affect the residual at \(x=2\)?
Why is a fixed cutoff of five response units only an illustrative rule rather than a universal decision rule?
Can a case have an unusual predictor value but a zero response residual? Explain using the third worked example.