Internally and Externally Studentized Residuals

Why Studentize a Residual?

A raw regression residual is the difference between an observed response and its fitted value. Its size alone does not indicate how unusual it is: residuals have different variances depending on their leverage, and the overall error variance must also be estimated. Studentized residuals put a residual on a scale that accounts for both features.

The distinction between internally and externally studentized residuals is the source of an important diagnostic difference. Internal studentization estimates error variance using all observations, including the one being assessed. External studentization estimates it after deleting that observation. When an observation has a large residual, including it in the variance estimate can inflate the estimated scale and make that residual look less unusual.

Definition: Consider the linear regression model \( \mathbf{y}=X\boldsymbol{\beta}+\boldsymbol{\varepsilon} \), where \(X\) has full column rank and \(p\) columns, including any intercept. Let \(n\) be the number of observations, \(e_i=y_i-\hat{y}_i\) the ordinary residual, and \(h_{ii}\) the \(i\)th diagonal entry of the hat matrix. Write \(SSE=\sum_{j=1}^n e_j^2\), \(s^2=SSE/(n-p)\), and \(s_{(i)}^2\) for the residual mean square from fitting the model without observation \(i\). The internally studentized residual is \(r_i=e_i/[s\sqrt{1-h_{ii}}]\); the externally studentized residual is \(t_i=e_i/[s_{(i)}\sqrt{1-h_{ii}}]\).

These formulas assume the ordinary least squares model has a common error variance. For exact distributional results, also assume independent normal errors with mean zero and variance \( \sigma^2 \), conditional on the design matrix. The regression fit with observation \(i\) deleted must remain estimable; in particular, the remaining design matrix must have full column rank. The formulas can still be calculated without normal errors, but the exact \(t\)-distribution statement below then need not hold.

Internal Studentization

The ordinary residual variance depends on leverage. Under the model, \( \operatorname{Var}(e_i\mid X)=\sigma^2(1-h_{ii}) \). Since \(s\) estimates \(\sigma\), dividing \(e_i\) by \(s\sqrt{1-h_{ii}}\) adjusts for the residual's leverage-dependent standard deviation and for the estimated error scale.

The internal residual uses \(s^2=SSE/(n-p)\), the residual mean square from the full fit. It is convenient because the full model supplies both \(e_i\) and \(s\). However, the observation being assessed contributes \(e_i^2\) to \(SSE\). A large residual therefore tends to increase the estimated variance used to standardize itself.

Worked Example: Calculating an Internally Studentized Residual

A hypothetical insurer models claim-processing time using a regression with an intercept and one predictor. There are \(n=8\) claims, the full-model \(SSE\) is 24, and for claim \(i\), \(e_i=3\) days and \(h_{ii}=0.25\). Calculate its internally studentized residual. Name the method and identify the mistake it avoids.

Method. Use the full-model residual mean square and adjust the residual standard deviation by the leverage factor.

The model has \(p=2\) coefficients, so its residual degrees of freedom are \(n-p=8-2=6\). Thus

s^2=\frac{24}{6}=4,\qquad s=2.

The leverage-adjusted residual standard deviation is

s\sqrt{1-h_{ii}}=2\sqrt{1-0.25} =2\sqrt{0.75} =2(0.8660)=1.7321.

Therefore

r_i=\frac{3}{1.7321}=1.7321.

To four decimal places, the internally studentized residual is 1.7321. The calculation avoids treating the raw residual of 3 days as if every claim had the same residual standard deviation. It does not remove the claim's contribution to the estimate \(s^2\).

Deleting an Observation and Estimating the Error Variance

To form an externally studentized residual, refit the regression without observation \(i\). The residual for that observation relative to the deleted fit is not simply \(e_i\): it is \(e_i/(1-h_{ii})\). The deletion also changes the residual sum of squares and the residual degrees of freedom. The resulting error-variance estimate is

s_{(i)}^2 = \frac{SSE-\dfrac{e_i^2}{1-h_{ii}}}{n-p-1}.

The denominator is \(n-p-1\) because the deleted fit uses \(n-1\) observations and still estimates \(p\) coefficients. The numerator removes the amount \(e_i^2/(1-h_{ii})\) from the full-model \(SSE\). This deletion identity is essential: subtracting only \(e_i^2\) would not account for how the fitted coefficients change when the observation is removed.

Derivation: Let \(SSE_{(i)}\) be the residual sum of squares from fitting the model without observation \(i\). The deleted-fit prediction error for observation \(i\) is \(e_i/(1-h_{ii})\). The least-squares sum-of-squares decomposition gives \(SSE=SSE_{(i)}+e_i^2/(1-h_{ii})\): the full fit's improvement over the deleted fit is exactly the squared deleted prediction error weighted by \(1-h_{ii}\). Rearranging gives \(SSE_{(i)}=SSE-e_i^2/(1-h_{ii})\). Dividing by the deleted fit's \(n-p-1\) residual degrees of freedom gives the displayed formula for \(s_{(i)}^2\).

The external residual is then

t_i=\frac{e_i}{s_{(i)}\sqrt{1-h_{ii}}}.

Equivalently, studentize the deleted prediction error \(e_i/(1-h_{ii})\) using its standard error \(s_{(i)}/\sqrt{1-h_{ii}}\). The two factors of \(1-h_{ii}\) simplify to the displayed expression.

Worked Example: Deleting a Claim Before Estimating Variance

In a hypothetical regression of claim-review cost on two predictors and an intercept, \(n=12\), \(p=3\), and the full-model \(SSE=54\). For one claim, \(e_i=4\) thousand dollars and \(h_{ii}=1/3\). Calculate \(s_{(i)}^2\) and the externally studentized residual. Name the method and the mistake it avoids.

Method. Use the case-deletion identity to obtain the deleted-fit sum of squares, then standardize using the deleted-fit variance estimate.

First, the amount removed from \(SSE\) is

\frac{e_i^2}{1-h_{ii}} =\frac{16}{1-\frac13} =\frac{16}{\frac23} =24.

Thus \(SSE_{(i)}=54-24=30\). The deleted fit has \(12-1-3=8\) residual degrees of freedom, so

s_{(i)}^2=\frac{30}{8}=3.75,\qquad s_{(i)}=\sqrt{3.75}=1.9365.

The leverage-adjusted standard error is

s_{(i)}\sqrt{1-h_{ii}} =1.9365\sqrt{\frac23} =1.5811.

Therefore \(t_i=4/1.5811=2.5298\). To four decimal places, \(s_{(i)}^2=3.7500\) and the externally studentized residual is 2.5298. The calculation avoids subtracting just \(e_i^2=16\) from the full \(SSE\); the correct deletion amount is 24.

How the Two Residuals Are Related

The internal and external quantities can be compared directly, without first calculating \(SSE_{(i)}\) from scratch. Let \(\nu=n-p\), the full fit's residual degrees of freedom. Since \(e_i^2/(1-h_{ii})=r_i^2s^2\), the deletion formula gives

s_{(i)}^2 = \frac{\nu s^2-r_i^2s^2}{\nu-1} = s^2\frac{\nu-r_i^2}{\nu-1}.

Substitute \(s_{(i)}=s\sqrt{(\nu-r_i^2)/(\nu-1)}\) into the external-residual formula. The common factor \(s\sqrt{1-h_{ii}}\) cancels, leaving

t_i=r_i\sqrt{\frac{\nu-1}{\nu-r_i^2}} = r_i\sqrt{\frac{n-p-1}{n-p-r_i^2}}.

This relation also explains why the two values often differ. If \(r_i^2>1\), deleting the observation reduces the estimated error variance, and the external residual has greater absolute magnitude than the internal residual. If \(r_i^2<1\), deletion can instead increase the variance estimate, making the external residual smaller in magnitude. The relation requires a positive deleted residual sum of squares and a valid deleted fit.

Worked Example: Comparing Internal and External Values

A hypothetical regression of monthly policy-service expenses on one predictor has \(n=10\), \(p=2\), \(SSE=32\), and, for one month, \(e_i=4\) thousand dollars and \(h_{ii}=0.20\). Calculate both studentized residuals and check their relationship. Name the method and the mistake it avoids.

Method. Calculate the internal residual from the full fit, then delete the month to obtain the external residual. This comparison avoids assuming that the same variance estimate belongs in both measures.

The full model has \(\nu=10-2=8\) residual degrees of freedom. Hence \(s^2=32/8=4\), and \(s=2\). The leverage-adjusted denominator is

2\sqrt{1-0.20}=2\sqrt{0.80}=1.7889.

Thus \(r_i=4/1.7889=2.2361\), with \(r_i^2=5\). For the deleted fit,

SSE_{(i)}=32-\frac{16}{0.80}=32-20=12, \qquad s_{(i)}^2=\frac{12}{10-2-1}=\frac{12}{7}=1.7143.

The external denominator is

\sqrt{\frac{12}{7}}\sqrt{0.80} =\sqrt{\frac{48}{35}} =1.1711,

so \(t_i=4/1.1711=3.4157\). The relation gives the same result:

2.2361\sqrt{\frac{7}{8-5}} =2.2361\sqrt{\frac73} =3.4157.

To four decimal places, the internal value is 2.2361 and the external value is 3.4157. The increase occurs because including this month's relatively large residual raises the full-fit variance estimate.

Interpretation and Common Traps

Under independent normal errors with common variance, \(t_i\) has a Student \(t\) distribution with \(n-p-1\) degrees of freedom when the model and design are fixed and observation \(i\) is assessed under the model. This exact result is one reason external studentization is useful for screening responses. Internal residuals are computationally direct, but their shared variance estimate includes the case being screened, so they do not have that same exact \(t\) distribution.

Studentized residuals are diagnostic measures, not automatic instructions to delete data. A large value can prompt a check for data-entry or measurement errors, an unusual but valid case, or model misspecification. The observation's leverage and the broader regression context also matter; residual size alone does not establish that a point is influential or erroneous.

Using \(n-p\) degrees of freedom after deletion. The deleted fit has \(n-1\) observations and \(p\) fitted coefficients, so its residual degrees of freedom are \(n-p-1\).
Subtracting \(e_i^2\) from \(SSE\). The correct reduction is \(e_i^2/(1-h_{ii})\), which accounts for the changed fitted coefficients.
Putting the wrong variance estimate in the formula. Internal studentization uses \(s\); external studentization uses \(s_{(i)}\).
Forgetting leverage adjustment. Both formulas include \(\sqrt{1-h_{ii}}\). A raw residual divided only by \(s\) is not either of these studentized residuals.
Treating a screening flag as proof. A diagnostic value points to a case for investigation; it does not by itself justify exclusion.

Key takeaway: Internal studentization uses the full-fit variance estimate, while external studentization recalculates that estimate after deleting the case. The external measure accounts for how the case affects the scale used to judge its own residual, which makes it especially useful for response-outlier screening.

Check Your Understanding

For each problem, use the stated regression summaries and keep the deleted-fit degrees of freedom distinct from the full-fit degrees of freedom.

A regression has \(n=14\), \(p=3\), and \(SSE=44\). For one observation, \(e_i=3\) and \(h_{ii}=0.25\). Find its internally studentized residual, rounded to four decimals. (Answer: \(1.7321\))
A regression has \(n=15\), \(p=4\), and \(SSE=66\). For an observation, \(e_i=3\) and \(h_{ii}=0.50\). Find the deleted-fit variance estimate \(s_{(i)}^2\). (Answer: \(4.8000\))
A regression has \(n=11\), \(p=2\), and \(SSE=36\). For an observation, \(e_i=3\) and \(h_{ii}=0.50\). Find its external studentized residual, rounded to four decimals. (Answer: \(2.8284\))
A regression has \(n=20\), \(p=5\), and an internally studentized residual \(r_i=2\). Use the relationship between the two residuals to find \(t_i\), rounded to four decimals. (Answer: \(2.2588\))
In a normal-error regression with \(n=18\) observations and \(p=4\) coefficients, what degrees of freedom apply to the exact Student \(t\) distribution of an externally studentized residual? (Answer: \(18-4-1=13\) degrees of freedom)

What You'll Learn