Externally Studentized Residuals and Outlier Checks

Why Refit When Checking an Unusual Response?

A regression residual is the difference between an observed response and its fitted value. A large residual can flag a response that is unusual for its predictor values. But the ordinary residual variance estimate uses every observation, including the one being checked. If that response is unusually far from the fitted line, it can inflate the estimated variance and make its own residual look less unusual.

An externally studentized residual addresses this problem by deleting observation $i$, fitting the model to the remaining observations, and estimating the error variance from that reduced fit. The deleted observation is then compared with the prediction from a model that did not use its response. This gives a useful individual outlier check under the regression assumptions. It does not, by itself, establish that a data value is wrong or that the observation has an influential effect on the fitted model.

Definition and Reference Distribution

Assume a linear regression model with an intercept and $k$ predictors. There are $n$ observations, the design matrix has full column rank, and the errors are independent and normally distributed with mean zero and a common variance $\sigma^2$. Treat the predictor values as fixed for this calculation. For the exact t reference distribution, the model must also be correctly specified.

For observation $i$, let $\widehat{y}_{i(i)}$ be the fitted response at its predictor values from a model refitted without observation $i$. Let $SSE_{(i)}$ be the residual sum of squares from that deleted fit, and let $c_i$ be the predictor-based part of the deleted-fit prediction variance: $c_i=1+\mathbf{x}_i^\mathsf{T}(\mathbf{X}_{(i)}^\mathsf{T}\mathbf{X}_{(i)})^{-1}\mathbf{x}_i$. Here, $\mathbf{x}_i$ is the design vector for observation $i$, including its intercept entry, and $\mathbf{X}_{(i)}$ is the design matrix with row $i$ removed. This definition requires the deleted design matrix to have full column rank.

Definition: The deleted residual is $e_{i(i)}=y_i-\widehat{y}_{i(i)}$. The deleted error variance estimate is $s^2_{(i)}=SSE_{(i)}/(n-k-2)$, because the deleted fit has $n-1$ observations and estimates $k+1$ regression coefficients. The externally studentized residual is $$ t_i=\frac{e_{i(i)}}{s_{(i)}\sqrt{c_i}}. $$

The factor $c_i$ accounts for uncertainty in predicting at the predictor values of the deleted observation. The $1$ in its definition accounts for the variation of a new response around its mean; the remaining term accounts for estimating the regression coefficients from the reduced data. Thus the denominator is the standard error for a future response at those predictor values, estimated without using $y_i$.

Under the stated assumptions, $t_i$ has a $t$ distribution with $n-k-2$ degrees of freedom when observation $i$ follows the model. This is a marginal reference distribution for checking that particular observation. It is not a guarantee that the largest residual among many checked observations follows the same t distribution.

Why the t Reference Distribution Applies

The deleted fit uses responses other than $y_i$, so its coefficient estimates and variance estimate do not use the response being tested. Conditional on the predictor values, the deleted residual is a difference between $y_i$ and a fitted mean based on the remaining responses. Its variance is $\sigma^2 c_i$. The deleted residual sum of squares, divided by $\sigma^2$, has a chi-square distribution with $n-k-2$ degrees of freedom and is independent of that prediction error under the normal linear model.

Consequently, the numerator divided by its true standard deviation has a standard normal distribution, while the estimated variance supplies an independent chi-square term. Their ratio has the t distribution. In particular,

\frac{e_{i(i)}}{\sigma\sqrt{c_i}}\sim N(0,1), \qquad \frac{SSE_{(i)}}{\sigma^2}\sim\chi^2_{n-k-2}, \qquad t_i\sim t_{n-k-2}.

This justification depends on the normal, independent, constant-variance error model and a full-rank deleted design. If the model assumptions are seriously violated, the t-based outlier assessment need not be reliable.

Calculating an External Residual by Refitting

The most direct calculation is to refit the model without the candidate response, predict at its predictor values, and calculate the deleted fit’s variance estimate and prediction factor. The following example shows each step for simple linear regression.

Worked Example: Checking the Last Response in a Small Regression

In a hypothetical study of five machines, an analyst models a daily output score $y$ as a linear function of operating time $x$. The observations are $(0,1),(1,2),(2,2),(3,4),(4,6)$. Assume independent normal errors with mean zero and constant variance, and a correctly specified linear model with an intercept. Check the last response using a two-sided individual 5% test.

Method. Delete $(4,6)$ and fit the regression to the other four observations. For the reduced data, $\sum x=6$, $\sum y=9$, $\sum x^2=14$, and $\sum xy=18$. Thus $\bar{x}=1.5$, $\bar{y}=2.25$, $S_{xx}=14-6^2/4=5$, and $S_{xy}=18-(6)(9)/4=4.5$.

The deleted-fit slope is $4.5/5=0.9$, and its intercept is $2.25-(0.9)(1.5)=0.9$. At $x=4$, the predicted response is $0.9+(0.9)(4)=4.5$. Therefore the deleted residual is $e_{i(i)}=6-4.5=1.5$.

The fitted values for the four retained observations, at $x=0,1,2,3$, are $0.9,1.8,2.7,3.6$. Their residuals are $0.1,0.2,-0.7,0.4$, so $SSE_{(i)}=0.1^2+0.2^2+(-0.7)^2+0.4^2=0.01+0.04+0.49+0.16=0.70$. There are $4-2=2$ residual degrees of freedom in the deleted fit, giving $s^2_{(i)}=0.70/2=0.35$.

For simple linear regression, the deleted prediction factor at $x_i=4$ is $c_i=1+1/4+(4-1.5)^2/5=1+0.25+1.25=2.50$. The standard error is therefore $\sqrt{0.35(2.50)}=\sqrt{0.875}=0.9354$.

The externally studentized residual is $t_i=1.5/0.9354=1.604$, with $4-2=2$ degrees of freedom. The two-sided 5% critical value is approximately $4.303$. Since $|1.604|<4.303$, this observation is not unusual at the 5% level using this individual check. The calculation avoids the mistake of using the full-data variance estimate after including the response being checked.

A Shortcut From an Internally Standardized Residual

If the internally standardized residual $r_i$ and the full-model residual degrees of freedom $\nu=n-k-1$ are already available, the external residual can also be calculated without repeating all the deleted-fit arithmetic. This conversion is useful, but a direct deleted fit remains a reliable way to understand what is being estimated.

Result: Provided the deleted fit is estimable and the relevant residual is not at a boundary case, $$ t_i=r_i\sqrt{\frac{\nu-1}{\nu-r_i^2}}, \qquad \text{with } \nu-1=n-k-2 \text{ degrees of freedom}. $$

To see why the conversion holds, let $s^2=SSE/\nu$ be the full-model error variance estimate and write the internally standardized residual as $r_i=e_i/[s\sqrt{1-h_{ii}}]$, where $e_i$ is the full-fit residual and $h_{ii}$ is the corresponding diagonal element of the projection matrix. The least-squares deletion identity is $SSE_{(i)}=SSE-e_i^2/(1-h_{ii})$. Since $e_i^2/(1-h_{ii})=r_i^2s^2=r_i^2SSE/\nu$, it follows that $SSE_{(i)}=SSE(\nu-r_i^2)/\nu$. Hence $s^2_{(i)}=SSE_{(i)}/(\nu-1)=s^2(\nu-r_i^2)/(\nu-1)$. The deleted residual equals $e_i/(1-h_{ii})$, and the deleted prediction factor equals $1/(1-h_{ii})$. Substituting these relationships into the definition of $t_i$ gives the stated conversion.

Worked Example: Converting a Standardized Residual

A correctly specified simple linear regression has $n=12$ observations. For one case, the internally standardized residual is $r_i=2.0$. Assume independent normal errors with constant variance. Convert it to an externally studentized residual and assess it using a two-sided individual 5% check.

Method. There is one predictor, so the full-model residual degrees of freedom are $\nu=12-1-1=10$. The external residual has $10-1=9$ degrees of freedom. Apply the conversion: $t_i=2.0\sqrt{9/(10-2.0^2)}=2.0\sqrt{9/6}=2.0\sqrt{1.5}=2.449$.

The two-sided 5% critical value for 9 degrees of freedom is approximately $2.262$. Since $|2.449|>2.262$, this case is unusual at the 5% level when considered by itself. This conclusion is an individual check, not a correction for inspecting all 12 responses. The conversion avoids treating $r_i$ as though it already had the external residual’s t distribution.

Many Checks and Common Interpretation Errors

If each of $n$ observations is checked separately at level 5%, the probability of at least one false flag can exceed 5%. The tests are not generally independent, but a Bonferroni adjustment controls the familywise error rate without requiring independence: for $n$ two-sided checks, compare each individual p-value with $\alpha/n$, or use a two-sided t critical value with tail probability $\alpha/(2n)$. This is more conservative than an individual check. The appropriate number of checks and the purpose of the analysis should be decided before interpreting a collection of flags.

An external residual identifies a response that is unusual relative to the fitted model at its predictor values. It does not establish why the response is unusual. Check the recorded value and the data-collection process, then consider whether the assumed regression model is appropriate. A large external residual also does not by itself measure how much the regression fit changes when the observation is removed; that is a separate question.

Worked Example: Individual Flag Versus Multiple Checks

An analyst fits a simple linear regression to 20 hypothetical measurements and examines all 20 externally studentized residuals. One case has $t_i=2.449$ with 17 degrees of freedom, and its two-sided individual p-value is approximately $0.025$. The analyst wants to control the familywise error rate at 5% using Bonferroni.

Method. The per-case p-value threshold is $0.05/20=0.0025$. The observed p-value $0.025$ is larger than $0.0025$, so this case does not pass the Bonferroni-adjusted check. Equivalently, each two-sided tail must have probability at most $0.05/(2\cdot20)=0.00125$.

The case would pass a 5% individual check because $0.025<0.05$, but not the stated familywise procedure. This does not show that the response is ordinary; it shows that evidence from one moderate residual is less compelling when 20 cases were searched. The method avoids reporting an unadjusted individual result as though only one observation had been examined.

Practical Calculation Checklist

State the model and the case being checked.
Confirm the regression assumptions and that removing the observation leaves a full-rank design.

Refit without that case.
Calculate its deleted prediction and the deleted residual $e_{i(i)}$.

Estimate deleted-fit uncertainty.
Use $s^2_{(i)}=SSE_{(i)}/(n-k-2)$ and the deleted prediction factor $c_i$.

Calculate and interpret the t statistic.
Divide by $s_{(i)}\sqrt{c_i}$, use $n-k-2$ degrees of freedom, and account for multiple checks if applicable.

Key takeaway: An externally studentized residual compares an observation with a prediction from a fit that excludes its response, using a variance estimate from that deleted fit. Its t reference is appropriate for an individual check under the normal linear model; searching many cases calls for a multiple-check adjustment.

Check Your Understanding

For each problem, use the stated regression assumptions and decide whether the requested calculation supports an individual flag unless a multiple-check adjustment is specified.

A simple linear regression has $n=9$. For one deleted case, $e_{i(i)}=-6$, $SSE_{(i)}=24$, and $c_i=1.5$. Calculate the externally studentized residual and its degrees of freedom. (Answer: $t_i=-2.449$, with 6 degrees of freedom.)
A regression with three predictors and an intercept has $n=15$. For a deleted case, $e_{i(i)}=5$, $SSE_{(i)}=90$, and $c_i=4/3$. Calculate the external residual and its degrees of freedom. (Answer: $t_i=1.443$, with 10 degrees of freedom.)
A regression has full-model residual degrees of freedom $\nu=14$, and a case has internally standardized residual $r_i=2.4$. Convert it to an external residual and give its reference degrees of freedom. (Answer: $t_i=2.4\sqrt{13/(14-2.4^2)}=3.015$, with 13 degrees of freedom.)
An analyst examines 20 two-sided externally studentized residuals and wants a Bonferroni familywise error rate of 0.05. What is the per-case p-value threshold? (Answer: $0.05/20=0.0025$.)
A simple linear regression has $n=10$, and one case has $r_i=1.8$. Convert it to an external residual and assess it at the individual two-sided 5% level, given that the 7-degree-of-freedom critical value is 2.365. (Answer: $t_i=1.8\sqrt{7/(8-1.8^2)}=2.183$, with 7 degrees of freedom; it does not exceed 2.365.)

What You'll Learn