Why Test for Outlying Responses?
A response can be unusual relative to a regression model even when the predictors themselves are not unusual. To assess this, compare an observation’s residual with the residual variation estimated from the other observations. This comparison is the purpose of an externally studentized residual: “externally” means that the observation being assessed is left out when estimating the residual variance.
A large absolute externally studentized residual is evidence that the response is unusual conditional on the specified predictors and model. It is a diagnostic signal, not proof of an error or a direction to delete the observation. Also, screening many observations requires accounting for the opportunity to find a large residual by chance.
Degrees of Freedom and the Rank Condition
The full-data fit has \(n-q\) residual degrees of freedom. After one observation is removed, there are \(n-1\) observations and still \(q\) fitted coefficients, so the deleted fit has
residual degrees of freedom. In particular, the deleted residual mean square is
where \(\operatorname{SSE}_{(i)}\) is the residual sum of squares from the fit omitting observation \(i\).
It is not enough that the original design matrix has full rank. The design must remain full rank after deletion. For a full-rank \(X\), this condition is equivalent to \(h_{ii}<1\). If \(h_{ii}=1\), the deleted fit cannot estimate all \(q\) coefficients; the displayed statistic is then undefined. A positive deleted residual mean square is also needed so that its square root is a valid denominator.
Why the Statistic Has a \(t\) Reference Distribution
The reference distribution follows from normal errors and the way the residual variance is estimated. Under the model, \(e_i\) has variance \(\sigma^2(1-h_{ii})\). Thus, when \(h_{ii}<1\),
The variance estimate \(s_{(i)}^2\) comes only from the regression fit that excludes observation \(i\). Under independent normal errors, it is independent of the standardized prediction error above, and
A standard normal variable divided by the square root of an independent chi-square variable divided by its degrees of freedom has a \(t\) distribution. Substituting the two quantities gives
This justification depends on the stated regression assumptions. If errors are not normally distributed, for example, the \(t\) reference distribution need not be exact. The statistic remains useful as a diagnostic, but its nominal p-value should not be treated as an exact test probability.
Testing One Response and Screening Many
For a two-sided test of whether observation \(i\) has an unusual response, use the null distribution \(t_{n-q-1}\). The p-value is the probability, under that distribution, of a value at least as far from zero as the observed statistic:
where \(T_{n-q-1}\) denotes a random variable with a \(t\) distribution with \(n-q-1\) degrees of freedom. A small p-value indicates a response that would be uncommon under the model, if the model and assumptions are appropriate.
When \(m\) observations are screened, testing each at level \(\alpha\) increases the chance of at least one false positive. The Bonferroni procedure controls the familywise probability of one or more false rejections by testing each observation at level \(\alpha/m\). Equivalently, compare each two-sided p-value with \(\alpha/m\), or compare each absolute residual with
This control does not require the tests to be independent. It can be conservative, particularly when many tests are performed. The threshold is for screening: crossing it flags an observation for investigation, not automatic removal from the data.
Worked Example: A Single Residual in a Multiple-Observation Screen
A hypothetical portfolio model has \(n=12\) observations and \(q=2\) fitted coefficients, including the intercept. For one observation, \(e_i=6\), \(h_{ii}=0.10\), and the deleted fit has \(\operatorname{SSE}_{(i)}=60\). Assess the residual using a two-sided test, and then decide whether it is notable in a Bonferroni screen of all 12 observations at familywise level \(0.05\).
Method. Calculate the deleted residual mean square, the externally studentized residual, and its individual p-value. Then apply the multiple-testing threshold.
The deleted degrees of freedom are
Therefore,
Since \(h_{ii}=0.10<1\), the denominator is defined. More directly,
Using a \(t\) distribution with 9 degrees of freedom, the two-sided p-value is approximately \(0.03679\). This is below \(0.05\), so it would be significant in an isolated two-sided test at the 5% level.
For 12 simultaneous tests, the Bonferroni per-test level is
The equivalent two-sided critical value is \(t_{1-0.05/(2\cdot12),9}=t_{0.997917,9}\approx3.808\). Since \(|2.4495|<3.808\), this observation is not flagged by the Bonferroni screen. The individual test and the familywise screen answer different questions.
Check and common mistake. The calculation uses the deleted-fit residual degrees of freedom, 9, rather than the full-fit residual degrees of freedom, 10. Comparing the statistic only with an unadjusted 5% cutoff would also fail to account for testing 12 observations.
Worked Example: Bonferroni Screening of Twenty Responses
A hypothetical regression has \(n=20\) observations and \(q=3\) fitted coefficients. All 20 externally studentized residuals are screened at familywise level \(0.05\). One observation has \(t_i=3.40\). Should it be flagged by the Bonferroni procedure?
Method. Find the deleted-fit degrees of freedom and the Bonferroni-adjusted two-sided cutoff, then compare absolute values.
For each observation the reference degrees of freedom are
The per-test level is
For a two-sided test, the upper-tail probability used for the critical value is \(0.0025/2=0.00125\). Thus the cutoff is \(t_{1-0.00125,16}=t_{0.99875,16}\approx3.581\). Since
the residual is not flagged by this familywise screen. It would exceed the unadjusted two-sided 5% cutoff, approximately \(t_{0.975,16}=2.120\), but that cutoff does not control the familywise error rate across 20 tests.
Check and common mistake. The two-sided Bonferroni cutoff uses \(\alpha/(2m)\) in the upper tail, not \(\alpha/m\). The per-test two-sided significance level is \(0.0025\), while the corresponding upper-tail probability is \(0.00125\).
Worked Example: Computing a Non-Significant Studentized Residual
In a hypothetical model with \(n=10\) observations and \(q=2\) fitted coefficients, an observation has \(e_i=4\), \(h_{ii}=0.20\), and \(\operatorname{SSE}_{(i)}=60\). Calculate its externally studentized residual and decide whether it is significant in an individual two-sided test at level \(0.05\).
Method. Use the deleted residual mean square and compare the resulting statistic with a two-sided \(t\) reference distribution.
The degrees of freedom are
The denominator and statistic are
The two-sided p-value from a \(t\) distribution with 7 degrees of freedom is approximately \(0.1705\), which exceeds \(0.05\). The residual is not statistically significant in this individual test.
Check and common mistake. The deleted mean square is \(60/7\), not \(60/8\): omitting one observation leaves 9 observations and 2 fitted coefficients, hence 7 residual degrees of freedom. The p-value is approximately \(0.1705\), not \(0.05\).
Interpreting a Flagged Observation
An externally studentized residual evaluates the response after accounting for the fitted predictors. A large absolute value can prompt a review of the recorded response, the model specification, and the assumptions used for inference. It does not, by itself, establish why the observation differs from the fitted relationship.
A useful screening sequence is:
Confirm that deleting the observation leaves a full-rank design, equivalently \(h_{ii}<1\), and that the deleted residual mean square is positive.
Use \(t_i=e_i/[s_{(i)}\sqrt{1-h_{ii}}]\) and \(n-q-1\) degrees of freedom.
For one prespecified observation, use the chosen individual significance level. For a screen of \(m\) observations, adjust for the multiple tests.
A flag identifies an observation whose response merits review; it is not a decision about whether the observation belongs in the analysis.
The nominal \(t\) test assumes that the regression form and error assumptions are appropriate. If the same data were used to select the model or to search among many diagnostic choices, the simple reference probabilities may not fully account for that selection. A test result should therefore be interpreted alongside the model diagnostics and the context of the data.
Check Your Understanding
For each problem, use the stated regression assumptions and degrees of freedom. Answers include the key result for checking your work.
- A model has \(n=16\) observations and \(q=3\) fitted coefficients. For one observation, \(e_i=5\), \(h_{ii}=0.36\), and \(s_{(i)}^2=4\). Calculate \(t_i\) and decide whether it exceeds the supplied two-sided 5% critical value \(t_{0.975,12}=2.179\). (Answer: \(t_i=3.125\); it exceeds the critical value.)
- A fixed model is used to screen 8 observations at familywise level \(0.05\). What is the Bonferroni per-test level? (Answer: \(0.00625\).)
- A model has deleted-residual degrees of freedom 14, and five observations are screened at familywise level \(0.05\). One has \(|t_i|=2.90\). Given \(t_{0.995,14}\approx2.977\), is it flagged? (Answer: No; \(2.90<2.977\).)
- For a model with \(n=15\), \(q=3\), \(e_i=6\), \(h_{ii}=0.25\), and \(\operatorname{SSE}_{(i)}=88\), calculate \(s_{(i)}^2\) and \(t_i\). (Answer: \(s_{(i)}^2=8\), \(t_i\approx2.4495\), with 11 degrees of freedom.)
- The original design has full column rank, but deleting observation \(i\) gives \(h_{ii}=1\). Can its externally studentized residual be calculated using the stated formula? (Answer: No; the deleted design is rank deficient and the denominator factor \(\sqrt{1-h_{ii}}\) is zero.)