Why Standardize a Regression Residual?
A raw residual is the difference between an observed response and the response predicted by a fitted regression line. Its size is measured in the response's units. A residual of 5 dollars may be substantial in one setting and negligible in another; even in the same regression, the variability of residuals can depend on where an observation lies in the predictor range. An internally standardized residual accounts for both the estimated error scale and the expected variability of that observation's fitted residual.
This calculation is a diagnostic, not a verdict. A large standardized residual indicates a response that merits closer examination relative to the fitted model. It does not by itself establish that the observation is erroneous, that the model is inappropriate, or that a formal statistical test has rejected a hypothesis.
Model, Residual Variability, and Leverage
Assume the simple linear regression model \(Y_i=\beta_0+\beta_1x_i+\varepsilon_i\), for \(i=1,\ldots,n\). The predictor values are treated as fixed, and the errors are independent, have mean zero and common variance \(\sigma^2\). For the usual normal-theory interpretation, also assume the errors are normally distributed. The least-squares fit estimates the intercept and slope, so the residual degrees of freedom are \(n-2\). We need \(n>2\), at least two distinct predictor values, and a nonzero residual standard error to calculate the statistic.
Write \(\bar{x}\) for the predictor mean and \(S_{xx}=\sum_{j=1}^n(x_j-\bar{x})^2\). In simple linear regression, the leverage of observation \(i\) is
The leverage is determined by the predictor values, not by the observed response. It is larger for predictor values farther from \(\bar{x}\). For a full-rank simple regression with \(n>2\), \(0\le h_{ii}<1\), so the residual adjustment has a positive denominator.
The fitted residuals do not all have variance \(\sigma^2\). The residual at observation \(i\) has variance \(\sigma^2(1-h_{ii})\). This is why standardizing a residual by \(s\) alone is not the internally standardized calculation.
Why the Leverage Adjustment Appears
A short matrix argument gives the variance adjustment. Let \(X\) be the design matrix with a column of ones and a column of predictor values. The projection matrix for least squares is \(H=X(X^\mathsf{T}X)^{-1}X^\mathsf{T}\). The residual vector is \(e=(I-H)\varepsilon\). Since \(H\) is symmetric and idempotent, \(I-H\) is also symmetric and idempotent. Under independent errors with common variance \(\sigma^2\), \(\operatorname{Var}(\varepsilon)=\sigma^2 I\), and hence
The \(i\)th diagonal element of \(H\) is \(h_{ii}\), so the \(i\)th diagonal element of \(I-H\) is \(1-h_{ii}\). Therefore \(\operatorname{Var}(e_i)=\sigma^2(1-h_{ii})\). Replacing the unknown \(\sigma\) with \(s\) gives the denominator \(s\sqrt{1-h_{ii}}\) in the definition. This also explains why a response with high leverage has less residual variability under the fitted model: the fitting process has used that response in estimating the line.
Calculation in Practice
A reliable calculation keeps the fitted residual, the regression standard error, and the leverage adjustment separate until the final division. The following sequence makes the degrees of freedom and the observation-specific adjustment explicit.
Calculate \(e_i=y_i-\hat{y}_i\), observed response minus fitted response.
Calculate \(s=\sqrt{\mathrm{SSE}/(n-2)}\), using the simple-regression residual degrees of freedom.
Calculate \(h_{ii}=1/n+(x_i-\bar{x})^2/S_{xx}\), then use \(\sqrt{1-h_{ii}}\).
Calculate \(r_i=e_i/[s\sqrt{1-h_{ii}}]\), then compare its magnitude with any screening guideline stated in the problem.
Worked Example: Standardizing a Residual at an Endpoint
A hypothetical study relates a device's operating temperature \(x\), in degrees, to its daily calibration deviation \(y\), in units. Five devices have data \((x,y)=(0,12),(1,12),(2,14),(3,18),(4,24)\). Find the internally standardized residual for the observation at \(x=4\), and decide whether it is flagged under the guideline \(|r_i|>2\).
Method. Fit the least-squares line, calculate the residual standard error and leverage at \(x=4\), and then standardize. The fitted line is \(\hat{y}=10+3x\): the means are \(\bar{x}=2\) and \(\bar{y}=16\), and the slope is \(30/10=3\), giving intercept \(16-3(2)=10\).
The fitted responses at \(x=0,1,2,3,4\) are \(10,13,16,19,22\). The residuals are \(2,-1,-2,-1,2\), so \(\mathrm{SSE}=4+1+4+1+4=14\). With \(n=5\),
Here \(S_{xx}=(-2)^2+(-1)^2+0^2+1^2+2^2=10\). At \(x_i=4\), the leverage is \(h_{ii}=1/5+(4-2)^2/10=0.6\). The residual is \(e_i=24-22=2\), so
Rounded to three decimal places, the internally standardized residual is \(1.464\). Its absolute value is below 2, so it is not flagged by this guideline. The check is consistent with the calculation: the residual is positive because the observed response is above its fitted value.
Worked Example: A Response Flagged for Investigation
A hypothetical sensor study records eight paired values. The predictor is a controlled setting \(x\); the response \(y\) is a measured output. For each observation, the table gives its residual from the least-squares line \(\hat{y}=20+2x\). Determine whether the response at \(x=1\) is flagged under \(|r_i|>2\).
| \(x\) | \(y\) | Residual \(e_i=y_i-\hat{y}_i\) |
|---|---|---|
| 1 | 29 | 7 |
| 2 | 20 | -4 |
| 3 | 23 | -3 |
| 4 | 26 | -2 |
| 5 | 29 | -1 |
| 6 | 32 | 0 |
| 7 | 35 | 1 |
| 8 | 38 | 2 |
First verify the stated fit: the residuals sum to zero, and their products with \(x-\bar{x}\), where \(\bar{x}=4.5\), also sum to zero. Thus the residuals are orthogonal to both regression columns, as least-squares residuals must be. The sum of squared residuals is \(49+16+9+4+1+0+1+4=84\). Therefore \(s^2=84/(8-2)=14\).
For \(x=1\), \(S_{xx}=42\) and \(h_{ii}=1/8+(1-4.5)^2/42=1/8+12.25/42=5/12\). The residual is \(e_i=29-(20+2(1))=7\). Hence
The absolute value exceeds 2, so the response is flagged for investigation. This does not prove the response is an error; it indicates that its residual is large relative to its fitted residual variability. The method avoids comparing the raw residual of 7 directly with 2, since those quantities are on different scales.
Worked Example: Using Regression Summaries and Rounding Correctly
A hypothetical study of a component's service time has \(n=15\), \(\bar{x}=10\), and \(S_{xx}=35\). At \(x_i=12\), the observed response is 10 and the fitted response is 18, so \(e_i=10-18=-8\). The regression output gives \(s^2=84\). Calculate the standardized residual and apply \(|r_i|>2\).
The leverage is \(h_{ii}=1/15+(12-10)^2/35=7/105+12/105=19/105\). Thus \(1-h_{ii}=86/105\), and the squared denominator is \(84(86/105)=68.8\). Therefore
Rounded to three decimal places, the residual is \(-0.964\), so it is not flagged. The sign is negative because the observed response is below the fitted response. Retaining the square root and rounding only at the end is important: the denominator is approximately \(8.295\), not \(68.8\).
Interpreting a Flag
A common exploratory guideline is to investigate observations with \(|r_i|>2\); values beyond 3 are often treated as especially notable. These are screening conventions, not universal cutoffs or automatic deletion rules. If a problem specifies a different rule, use that rule. A formal probability assessment requires more than mechanically comparing a residual with a threshold, and internally standardized residuals are not independent standard normal observations.
A flagged response should be checked in context: confirm the recorded value and units, examine whether the regression form is reasonable, and consider whether the assumed error variance is appropriate. If the residual spread changes across predictor values, the common-variance assumption used in the standardization may fail. A standardized residual can still be calculated, but its interpretation as a comparison with expected residual variability is then less reliable.
This diagnostic concerns unusual responses relative to the fitted line and its residual variability. It is distinct from deciding whether a predictor value is unusual, and a large standardized residual alone does not measure an observation's overall influence on the fitted regression.
Common Calculation Traps
- Omitting the leverage adjustment. Dividing \(e_i\) by \(s\) alone ignores the fact that the residual standard deviation is \(s\sqrt{1-h_{ii}}\). For example, if \(e_i=-4\), \(s=1.6\), and \(h_{ii}=0.10\), the incorrect calculation is \(-4/1.6=-2.500\). The correct calculation is \(-4/(1.6\sqrt{0.90})\approx-2.635\), which is flagged under \(|r_i|>2\).
- Reversing the residual sign. The definition is observed minus fitted, \(e_i=y_i-\hat{y}_i\). If \(y_i=10\) and \(\hat{y}_i=8+(45/82.5)(10)=148/11\), then \(e_i=10-148/11=-38/11\approx-3.455\), not \(+3.455\). The response lies below the fitted value.
- Using the wrong residual degrees of freedom. Simple linear regression estimates two coefficients, so \(s^2=\mathrm{SSE}/(n-2)\), not \(\mathrm{SSE}/n\) or \(\mathrm{SSE}/(n-1)\).
- Calling a flag proof of an outlier. A flag means investigate under the chosen screening convention. It does not justify deleting a data point without checking the data and model.
- Rounding intermediate values too early. Keep the exact leverage fraction and squared denominator through the calculation; round the final standardized residual to the requested precision.
Check Your Understanding
For each problem, calculate the requested internally standardized residual before applying the stated guideline. Round final residuals to three decimal places.
- A calibration regression has \(n=12\), \(s=3\), residual \(e_i=-4\), and leverage \(h_{ii}=0.20\). Calculate \(r_i\) and decide whether it is flagged under \(|r_i|>2\). (Answer: \(r_i\approx-1.491\); not flagged.)
- In a hypothetical equipment study, \(n=10\), \(\bar{x}=5\), \(S_{xx}=30\), \(x_i=8\), \(e_i=6\), and \(s=2.5\). Calculate leverage and the standardized residual, then apply \(|r_i|>2\). (Answer: \(h_{ii}=0.400\), \(r_i\approx3.098\); flagged for investigation.)
- A simple regression has \(n=9\), \(\mathrm{SSE}=56\), and, for one observation, \(x_i-\bar{x}=2\), \(S_{xx}=24\), and \(e_i=-5\). Calculate \(s\), leverage, and the standardized residual. Use \(|r_i|>2\). (Answer: \(s=\sqrt{8}\approx2.828\), \(h_{ii}=5/18\), \(r_i\approx-2.080\); flagged for investigation.)
- Five observations have \(x=1,2,3,4,5\) and responses \(y=7,6,10,10,15\). The fitted line is \(\hat{y}=4+2x\). Find the internally standardized residual for \(x=5\) and apply \(|r_i|>2\). (Answer: residuals are \(1,-2,0,-2,1\), so \(\mathrm{SSE}=10\), \(s=\sqrt{10/3}\), \(h_{ii}=0.600\), and \(r_i\approx0.866\); not flagged.)
- An observation has \(e_i=-4\), \(s=1.6\), and \(h_{ii}=0.10\). Calculate the internally standardized residual and decide whether it is flagged under \(|r_i|>2\). (Answer: \(r_i\approx-2.635\); flagged for investigation.)