Externally Studentized Deleted Residuals

Why Studentize a Residual Using a Deleted Fit?

An ordinary residual compares an observed outcome with the value fitted by a regression model. Its size alone is hard to assess: residuals have different variances depending on leverage, and the full-sample estimate of error variance includes the case being examined. An externally studentized deleted residual addresses both issues. It adjusts for leverage and estimates the error variance using a fit that omits the case.

The result helps identify an outcome that is unusual given its predictors and the other observations. It does not by itself tell us whether that case substantially changes the fitted model. That distinction matters: a case can have an unusual outcome without being influential, or have high leverage without an unusually large residual.

Definition and Model Assumptions

Suppose there are \(n\) observations and \(q\) predictors, with an intercept in the model. Write the model for observation \(i\) as

For exact t-based interpretations, assume the predictor values are fixed, the mean model is correctly specified, and the errors are independent with \(E(\varepsilon_i)=0\), common variance \(\sigma^2\), and a normal distribution. The design matrix must have full column rank both before and after case \(i\) is omitted. These are classical model assumptions; departures from them can affect the diagnostic and its reference distribution.

Fit the model to all observations by least squares. Let \(\hat{y}_i\) be the fitted value, \(e_i=y_i-\hat{y}_i\) the residual, and \(h_{ii}\) the hat value (leverage) for case \(i\). The full fit has \(n-q-1\) residual degrees of freedom. Now refit the model after omitting case \(i\). Let \(SSE_{(i)}\) be the residual sum of squares from that deleted fit. That fit has \(n-1\) observations and \(q+1\) estimated coefficients, so its residual degrees of freedom are \(n-q-2\).

The numerator is the residual from the full fit, while the scale estimate comes from a fit that excludes the case. Thus the case cannot directly inflate its own estimate of residual variance. The factor \(\sqrt{1-h_{ii}}\) accounts for the fact that residuals from the full fit have variance \(\sigma^2(1-h_{ii})\), rather than \(\sigma^2\), under the stated model.

Why the Deleted Statistic Has a t Reference Distribution

There is a useful prediction interpretation. Let \(\hat{y}_{i,(i)}\) be the fitted value for case \(i\)'s predictors from the model that omits case \(i\). The discrepancy \(y_i-\hat{y}_{i,(i)}\) compares the omitted outcome with a prediction made without using that outcome. It is often called a deleted prediction residual. Standard least-squares deletion algebra gives

where \(x_i\) is the row of predictors including the intercept, \(X_{(i)}\) is the design matrix with that row removed, and \(\mathsf{T}\) denotes transpose. The second expression is the variance multiplier for predicting a new outcome at \(x_i\) from the deleted fit. Dividing the deleted prediction residual by its estimated prediction standard deviation gives the same \(t_i\) as the definition above.

To see why the t distribution applies under the normal-error assumptions, the deleted fit depends only on observations other than \(i\). Its prediction error for observation \(i\) has mean zero and variance \(\sigma^2\) times the prediction variance multiplier. The deleted residual sum of squares, divided by \(\sigma^2\), has a chi-square distribution with \(n-q-2\) degrees of freedom. It is independent of that prediction error because the omitted observation's error is independent of the errors used in the deleted fit, and the normal least-squares residual component is independent of its fitted component. Therefore the standardized prediction error is a standard normal variable divided by the square root of an independent chi-square variable divided by its degrees of freedom. That ratio has a t distribution:

This is a model-based reference distribution, not a guarantee that a particular case is unusual. A large absolute value means the outcome is far from what the model and other observations would predict, relative to the estimated prediction error. If the normal-error or constant-variance assumptions are doubtful, the exact t calibration may not be reliable.

Worked Examples: Interpreting the Direction and Size

What This Diagnostic Does—and Does Not—Measure

The externally studentized deleted residual is designed to assess outcome unusualness conditional on the predictors. It uses a variance estimate that does not include the case and standardizes for leverage. A large absolute value points to an outcome that is far from its model-based prediction relative to the uncertainty in predicting that outcome.

It is not an influence measure. Influence concerns how much a case affects fitted quantities, such as regression coefficients or fitted values. A case can have a small externally studentized residual but high leverage; conversely, a large residual at modest leverage need not cause a large change in the fit. The statistic is useful as one diagnostic among several, not as a stand-alone rule for deleting observations.

A related distinction is that an internally studentized residual uses the full-fit residual standard deviation, which includes the case being examined. The externally studentized deleted residual replaces that scale estimate with \(s_{(i)}\), estimated after the case is omitted. This difference can be consequential when one case contributes substantially to the full residual sum of squares: the full-sample estimate may be inflated, making that case's internally studentized residual appear less extreme.

Common Mistakes

• Calling a large value “influential.” A large absolute \(t_i\) indicates an unusual outcome relative to the fitted model, not a large change in coefficients or predictions. Assess influence with diagnostics designed for changes in the fit.
• Using the full-fit error variance in the formula. That produces an internally studentized residual, not an externally studentized deleted residual. The scale in this statistic is \(s_{(i)}\), calculated from the fit omitting case \(i\).
• Ignoring leverage. Dividing \(e_i\) only by \(s_{(i)}\) omits the \(\sqrt{1-h_{ii}}\) adjustment and does not give the defined statistic.
• Interpreting a p-value as the chance the case is not unusual. The p-value is calculated under the model assumptions; it is not the probability that the null model is true or that the case is “normal.”
• Deleting every flagged case automatically. A flag is a reason to check the data, model, and context. It is not proof of an error, and removal can distort the analysis.