Internally Studentized Residuals

Why Adjust Residuals for Leverage?

A residual measures the difference between an observed response and its fitted value. But residuals from the same regression do not all have the same variability: an observation with greater leverage typically has a smaller residual variance. Comparing raw residuals alone can therefore give an uneven picture of how unusual different observations are. An internally studentized residual puts a residual on a scale that accounts for both this leverage difference and the estimated error spread.

The adjustment matters when examining regression diagnostics. It makes residuals dimensionless, so their sizes can be compared across observations measured in the same response variable. It does not, by itself, establish that a case is influential, identify a cause of unusual data, or provide a test of a model assumption.

Definition and Regression Assumptions

Consider a least-squares regression model with $n$ observations and $m$ estimated coefficients, including the intercept if there is one. Write $y_i$ for the observed response, $\hat{y}_i$ for its fitted value, and $e_i=y_i-\hat{y}_i$ for its residual. Let $h_{ii}$ be the leverage, or diagonal hat value, for observation $i$. The residual sum of squares is $SSE=\sum_{i=1}^n e_i^2$, and the residual standard error is $s=\sqrt{SSE/(n-m)}$, with $n-m$ residual degrees of freedom.

For the variance adjustment to have its usual interpretation, assume the regression mean model is correctly specified, the design values are fixed (or treated as fixed), and the errors have mean zero, common variance $\sigma^2$, and are uncorrelated. Thus, in vector notation, the model is $y=X\beta+\varepsilon$, with $\operatorname{Var}(\varepsilon)=\sigma^2 I$. Normal errors are not needed to define or calculate the quantity; they are an additional assumption when exact distributional results are sought.

Definition: The internally studentized residual for observation $i$ is $$ r_i=\frac{e_i}{s\sqrt{1-h_{ii}}}. $$ Here $e_i$ is the residual, $h_{ii}$ is that observation's leverage, and $s$ is estimated from the residuals of the full regression, including observation $i$. The denominator estimates the standard deviation of $e_i$. The result is a signed, dimensionless value.

The word internally indicates that the estimate $s$ comes from the full fit. Since $e_i$ and $s$ are measured in the response's units, their units cancel in $r_i$; the leverage term is also dimensionless. A positive value means the observed response is above its fitted value, and a negative value means it is below.

Why the Leverage Adjustment Appears

The leverage term follows from the variance of a least-squares residual. Let $H$ be the hat matrix, so the fitted response vector is $Hy$ and the residual vector is $e=(I-H)y$. For a full-rank least-squares fit, $H$ is symmetric and idempotent: $H^\mathsf{T}=H$ and $H^2=H$. Because $(I-H)X=0$, the model's mean contributes no residual, giving $e=(I-H)\varepsilon$.

Using the common error variance assumption, the residual covariance matrix is

\operatorname{Var}(e) =(I-H)\operatorname{Var}(\varepsilon)(I-H)^\mathsf{T} =\sigma^2(I-H)(I-H)^\mathsf{T} =\sigma^2(I-H).

The final equality uses the symmetry and idempotence of $I-H$. Its $i$th diagonal entry is $1-h_{ii}$, so

\operatorname{Var}(e_i)=\sigma^2(1-h_{ii}).

Replacing the unknown $\sigma$ with the full-fit residual standard error $s$ gives the estimated residual standard deviation $s\sqrt{1-h_{ii}}$, which is the denominator in $r_i$. This is the key justification for scaling by leverage: residuals with different $h_{ii}$ have different variances even under the same error variance.

Worked Example: Calculating from a Regression Summary

Suppose a hypothetical least-squares regression uses $n=20$ observations and estimates $m=3$ coefficients. Its $SSE$ is $153$ response-units squared. For one observation, the residual is $e_i=6.0$ response units and its leverage is $h_{ii}=0.20$. Calculate and interpret its internally studentized residual.

First, calculate the residual standard error using $n-m=17$ residual degrees of freedom:

s=\sqrt{\frac{153}{20-3}} =\sqrt{9} =3.00\ \text{response units}.

The leverage-adjusted estimated standard deviation of this residual is

s\sqrt{1-h_{ii}} =3.00\sqrt{1-0.20} =3.00\sqrt{0.80} \approx 2.683\ \text{response units}.

Therefore,

r_i=\frac{6.0}{2.683}\approx 2.236.

The observed response is above its fitted value, and its residual is about $2.24$ estimated leverage-adjusted residual standard deviations above zero. The value is dimensionless. It is not a probability, nor does this calculation alone establish that the observation is influential.

A Small Regression Example

The next hypothetical data set illustrates how leverage changes the scaling across cases. For a simple linear regression with an intercept and one predictor, there are $m=2$ estimated coefficients. The following five observations lie around the fitted line $\hat{y}=10+2x$.

Observation	$x_i$	$y_i$	$\hat{y}_i$	$e_i=y_i-\hat{y}_i$	$h_{ii}$
1	-2	7	6	1	0.60
2	-1	6	8	-2	0.30
3	0	12	10	2	0.20
4	1	10	12	-2	0.30
5	2	15	14	1	0.60

Worked Example: Studentizing Residuals in a Simple Regression

All values in the table are hypothetical. The line is the least-squares fit: the residuals sum to zero, and their products with the centered predictor values also sum to zero. The listed leverages follow $h_{ii}=1/5+(x_i-\bar{x})^2/\sum_j(x_j-\bar{x})^2$; here $\bar{x}=0$ and $\sum_j(x_j-\bar{x})^2=10$. The model has $n=5$ and $m=2$.

First compute the residual sum of squares and residual standard error:

SSE=1^2+(-2)^2+2^2+(-2)^2+1^2=14, \qquad s=\sqrt{\frac{14}{5-2}} =\sqrt{\frac{14}{3}} \approx 2.160.

For observation 1, the residual is $1$, and $h_{11}=0.60$. Thus

r_1=\frac{1}{2.160\sqrt{1-0.60}} =\frac{1}{2.160\sqrt{0.40}} \approx 0.732.

For observation 2, the residual is $-2$, and $h_{22}=0.30$:

r_2=\frac{-2}{2.160\sqrt{1-0.30}} =\frac{-2}{2.160\sqrt{0.70}} \approx -1.107.

Applying the same calculation to the remaining observations gives $r_3\approx 1.035$, $r_4\approx -1.107$, and $r_5\approx 0.732$. The positive values correspond to observations above the fitted line; the negative values correspond to observations below it. Observations 1 and 5 have the same raw residual as one another, but their leverage-adjusted standard deviation is smaller than it is for observation 2. The resulting values express each residual relative to its own estimated standard deviation rather than comparing raw distances alone.

Interpreting the Scale Carefully

An internally studentized residual answers a focused question: how large is this observation's residual compared with the estimated standard deviation of residuals at its leverage? Its sign gives the direction of the departure from the fitted value; its magnitude gives the departure on that adjusted scale. It can help identify cases worth checking, but it is only one diagnostic and does not say why a residual is large.

The quantity is not automatically a standard-normal score. The denominator $s$ is estimated from the same full regression, and residuals from a fitted model are related to one another. In particular, do not turn $r_i$ directly into a normal-theory p-value without a justified distributional result and an appropriate procedure. The next diagnostic in this sequence uses a different error-scale estimate; that is distinct from the full-fit estimate used here.

The formula also relies on the common-variance assumption for the variance expression $\sigma^2(1-h_{ii})$. If the errors have unequal variances or are correlated, this expression need not describe the residual variance. A leverage adjustment does not repair a misspecified mean model or a violated error structure.

Worked Example: A Residual in Response Units

A hypothetical regression predicts clinic waiting time in minutes from two predictors, with $n=12$ observations and $m=3$ fitted coefficients. Suppose $SSE=81$ minutes squared. One case has an observed waiting time $4.5$ minutes above its fitted value, so $e_i=4.5$ minutes, and its leverage is $h_{ii}=0.36$. Calculate the internally studentized residual.

There are $12-3=9$ residual degrees of freedom, so the residual standard error is

s=\sqrt{\frac{81}{9}}=3.00\ \text{minutes}.

The leverage-adjusted estimate of residual standard deviation is

s\sqrt{1-h_{ii}} =3.00\sqrt{1-0.36} =3.00(0.80) =2.40\ \text{minutes}.

It follows that

r_i=\frac{4.5\ \text{minutes}}{2.40\ \text{minutes}} =1.875.

This case's observed waiting time is above its fitted value, by $1.875$ estimated leverage-adjusted residual standard deviations. The units cancel. Dividing by $s$ alone would give $4.5/3.00=1.50$, which omits the adjustment for this case's leverage and is not the internally studentized residual.

Common Mistake

Wrong reasoning: “A residual of 6 minutes is twice as unusual as a residual of 3 minutes, because it is twice as large.” This compares raw differences without considering either the overall error scale or the cases' leverages.
Correction: Calculate $r_i=e_i/[s\sqrt{1-h_{ii}}]$ for each case. The denominator accounts for the estimated error scale and the residual variance adjustment associated with leverage. Then interpret the result as a signed, dimensionless scaled residual—not as a probability or a standalone measure of influence.

Key takeaway: Internally studentized residuals divide each full-fit residual by $s\sqrt{1-h_{ii}}$. The adjustment reflects the leverage-dependent residual variance under the usual least-squares assumptions; it does not automatically provide a standard-normal score, a p-value, or a verdict about influence.

Check Your Understanding

Use the definition and assumptions above to answer these questions.

In a regression with $n=18$ observations and $m=4$ fitted coefficients, what degrees of freedom are used to calculate $s$?
If a case has $e_i=-3$, $s=2$, and $h_{ii}=0.36$, calculate its internally studentized residual and interpret its sign.
Why does the formula divide by $s\sqrt{1-h_{ii}}$ rather than by $s$ alone?
What does “internally” mean in the definition of an internally studentized residual?
Why is it incorrect to treat an internally studentized residual as automatically having a standard-normal distribution?