Tutorials › VEE Mathematical Statistics › Cook’s Distance for Overall Influence

Home  /  Tutorials  /  VEE Mathematical Statistics / Prediction and Model Assessment

VEE Mathematical Statistics · Prediction and Model Assessment · Tutorial 785 of 880

Cook’s Distance for Overall Influence

Calculate Cook’s distance from a regression fit and interpret how much the fitted results may change when an observation is removed.

Intermediate 10 min read

What You'll Learn

  • Define Cook’s distance as a scaled measure of change in all fitted values after deleting one observation
  • Calculate Cook’s distance from the residual, leverage, and residual mean square
  • Use the number of regression parameters, including the intercept, in the scaling
  • Compare observations with different residual sizes and leverage values
  • Interpret common reference values as screening heuristics rather than strict tests

Why Measure Overall Influence?

A regression observation can affect the fitted model in more than one way. Its residual measures how far its observed response is from the fitted response, while its leverage reflects how unusual its predictor value is relative to the other predictor values. Cook’s distance combines these effects to summarize how much the whole set of fitted values would change if that observation were removed and the model refitted.

This is useful when assessing whether a reported regression result depends heavily on one data point. Cook’s distance is a measure of change, not a formal test that an observation is erroneous. Its value should be interpreted in the context of the model and the rest of the data.

Definition and Formula

Consider an ordinary least-squares linear regression with \(n\) observations and \(p\) fitted parameters, including the intercept. Write \(X\) for the \(n\)-by-\(p\) design matrix, \(\hat{\beta}\) for the fitted coefficient vector, and \(\hat{y}_i\) for the fitted response at observation \(i\). The residual is \(e_i=y_i-\hat{y}_i\). The hat matrix is \(H=X(X'X)^{-1}X'\), and the leverage of observation \(i\) is its diagonal entry \(h_{ii}\). The residual mean square is \(MSE=SSE/(n-p)\), where \(SSE=\sum_{j=1}^{n}e_j^2\).

Definition: Cook’s distance for observation \(i\) is the sum of squared differences between the full-data fitted values and the fitted values after observation \(i\) is deleted, divided by \(pMSE\). When the full design matrix has full column rank, \(n>p\), and deleting observation \(i\) leaves the model estimable, it is calculated as
$$ D_i = \frac{\displaystyle\sum_{j=1}^{n} \left(\hat{y}_j-\hat{y}_{j(i)}\right)^2}{p\,MSE} = \frac{e_i^2}{p\,MSE}\, \frac{h_{ii}}{(1-h_{ii})^2}, $$

Here \(\hat{y}_{j(i)}\) is the fitted value for case \(j\) from the model fitted without observation \(i\). In simple linear regression with an intercept, \(p=2\), and the leverage is \(h_{ii}=1/n+(x_i-\bar{x})^2/S_{xx}\), where \(S_{xx}=\sum_{j=1}^{n}(x_j-\bar{x})^2\). The formula does not apply as written if deletion makes the design matrix rank deficient; in that case the reduced model cannot estimate all the original parameters.

The formula uses the residual from the full-data fit and the full-data \(MSE\), not a new residual mean square from the deleted fit. A larger absolute residual increases \(D_i\); so does greater leverage, especially as \(h_{ii}\) approaches 1.

Why the Formula Measures Change

The equality between the deletion-based definition and the computational formula follows from the least-squares normal equations. Let \(x_i\) be row \(i\) of \(X\), written as a column vector, and let \(C=(X'X)^{-1}\). Deleting case \(i\) changes the cross-product matrix from \(X'X\) to \(X'X-x_ix_i'\), and changes \(X'y\) to \(X'y-x_i y_i\). The resulting deletion coefficient vector is

$$ \hat{\beta}_{(i)} = \hat{\beta} - \frac{C x_i e_i}{1-h_{ii}}, \qquad h_{ii}=x_i'C x_i. $$

To verify this identity, multiply its right-hand side by the deleted cross-product matrix. Since \(X'X\hat{\beta}=X'y\), \(x_i'\hat{\beta}=\hat{y}_i\), and \(e_i=y_i-\hat{y}_i\), the product is

$$ \begin{aligned} (X'X-x_ix_i')\left(\hat{\beta}-\frac{Cx_i e_i}{1-h_{ii}}\right) &=X'y-x_i\hat{y}_i-\frac{x_i e_i}{1-h_{ii}} +\frac{x_i h_{ii} e_i}{1-h_{ii}}\\ &=X'y-x_i\hat{y}_i-x_i e_i\\ &=X'y-x_i y_i. \end{aligned} $$

This is precisely the normal equation for the data with case \(i\) removed. The vector of changes in fitted values is therefore \(X(\hat{\beta}-\hat{\beta}_{(i)})\). Its squared length is

$$ \begin{aligned} \left\|X(\hat{\beta}-\hat{\beta}_{(i)})\right\|^2 &=\frac{e_i^2}{(1-h_{ii})^2} x_i'C X'X Cx_i\\ &=\frac{e_i^2 h_{ii}}{(1-h_{ii})^2}, \end{aligned} $$

because \(C(X'X)C=C\). Dividing this total squared change by \(pMSE\) gives Cook’s distance. The derivation also shows why a large residual alone does not determine the measure: the change is multiplied by a term involving leverage.

Calculating and Comparing Cook’s Distance

Worked Example: Comparing a High-Leverage Case With a Central Case

A hypothetical simple linear regression uses five observations with predictor values \(x=(-2,-1,0,1,2)\) and responses \(y=(3,1,6,3,7)\). Calculate Cook’s distance for the observation at \(x=2\) and compare it with the observation at \(x=0\).

Method. Fit the full regression, calculate each case’s residual and leverage, then apply the Cook’s distance formula with \(p=2\). This avoids treating a raw residual as the complete measure of influence.

The predictor mean is \(\bar{x}=0\), and \(S_{xx}=(-2)^2+(-1)^2+0^2+1^2+2^2=10\). The response mean is \(\bar{y}=4\). The fitted slope is \(\sum x_i y_i/S_{xx}=10/10=1\), and the intercept is 4. Thus \(\hat{y}=4+x\), with residuals \(e=(1,-2,2,-2,1)\). Their squared sum is \(SSE=1+4+4+4+1=14\), so \(MSE=14/(5-2)=14/3\).

For \(x=2\), leverage is \(h_{ii}=1/5+2^2/10=3/5\), and \(e_i=1\). Therefore

$$ D_{x=2} = \frac{1^2}{2(14/3)} \frac{3/5}{(1-3/5)^2} = \frac{3}{28}\cdot\frac{15}{4} = \frac{45}{112} \approx 0.4018. $$

For \(x=0\), \(h_{ii}=1/5\) and \(e_i=2\). Hence

$$ D_{x=0} = \frac{2^2}{2(14/3)} \frac{1/5}{(1-1/5)^2} = \frac{3}{7}\cdot\frac{5}{16} = \frac{15}{112} \approx 0.1339. $$

The central observation has the larger residual, but its Cook’s distance is smaller. The case at \(x=2\) has greater leverage, so deleting it changes the full set of fitted values more by this measure. As a check, the sum of squared fitted-value changes for \(x=2\) is \(pMSE D_i=(28/3)(45/112)=15/4=3.75\), which is positive and consistent with the definition.

Worked Example: A Large Residual at an Extreme Predictor Value

A hypothetical model has predictor values \(x=(-2,-1,0,1,2)\) and responses \(y=(8,7,8,11,16)\). Find Cook’s distance for the observation at \(x=2\), then compare it with the observation at \(x=0\).

Method. Calculate the full-data residuals and leverage values before applying the formula. This keeps the distinction between residual size and leverage visible.

The predictor mean is 0 and \(S_{xx}=10\). The response mean is 10, and \(\sum x_i y_i=(-2)(8)+(-1)(7)+0(8)+1(11)+2(16)=20\). Thus the fitted slope is \(20/10=2\), the intercept is 10, and the fitted line is \(\hat{y}=10+2x\). The residuals are \((2,-1,-2,-1,2)\), so \(SSE=4+1+4+1+4=14\) and \(MSE=14/3\).

At \(x=2\), \(h_{ii}=1/5+4/10=3/5\) and \(e_i=2\). Thus

$$ D_{x=2} = \frac{4}{28/3}\frac{3/5}{(2/5)^2} = \frac{3}{7}\cdot\frac{15}{4} = \frac{45}{28} \approx 1.6071. $$

At \(x=0\), \(h_{ii}=1/5\) and \(e_i=-2\), giving

$$ D_{x=0} = \frac{4}{28/3}\frac{1/5}{(4/5)^2} = \frac{3}{7}\cdot\frac{5}{16} = \frac{15}{112} \approx 0.1339. $$

The two observations have equal residual magnitudes, but the extreme predictor value has much greater leverage and a substantially larger Cook’s distance. The deletion-based definition gives a check: for \(x=2\), the sum of squared fitted-value changes is \((28/3)(45/28)=15\).

Worked Example: Equal Residuals, Different Leverage

A hypothetical simple regression uses \(x=(-1,0,1,2)\) and \(y=(3,2,3,6)\). Compare Cook’s distance for \(x=-1\) and \(x=0\), whose residuals have equal magnitudes.

Method. Fit the line and calculate the leverage for each case. This comparison isolates the effect of predictor location when the residual magnitudes are the same.

Here \(\bar{x}=0.5\) and \(\bar{y}=3.5\). The centered cross-product is \((-1.5)(-0.5)+(-0.5)(-1.5)+(0.5)(-0.5)+(1.5)(2.5)=5\), while \(S_{xx}=5\). The fitted slope is \(5/5=1\), and the intercept is \(3.5-(1)(0.5)=3\). The fitted values are \((2,3,4,5)\), giving residuals \((1,-1,-1,1)\). Thus \(SSE=4\) and \(MSE=4/(4-2)=2\).

For \(x=-1\), the leverage is \(1/4+(-1-0.5)^2/5=0.7\). For \(x=0\), it is \(1/4+(0-0.5)^2/5=0.3\). Since \(pMSE=2(2)=4\), the two distances are

$$ D_{x=-1} = \frac{1}{4}\frac{0.7}{(0.3)^2} = \frac{35}{18} \approx 1.9444, \qquad D_{x=0} = \frac{1}{4}\frac{0.3}{(0.7)^2} = \frac{15}{98} \approx 0.1531. $$

The residual magnitudes are both 1, but the observation at \(x=-1\) is farther from the predictor mean and has much greater leverage. The deletion-based check gives squared fitted-value changes of \(4(35/18)=70/9\) and \(4(15/98)=30/49\), respectively. These values agree with the direction and scale of the calculated distances.

Interpretation and Common Exam Trap

Cook’s distance is a comparative diagnostic: within the same fitted model, a larger value indicates a larger scaled change in the fitted values on deleting that case. Heuristics such as \(D_i>1\) or \(D_i>4/n\) are sometimes used to flag cases for further examination. They are not universal significance cutoffs, and neither one proves that a data point is invalid or that the regression model is wrong.

A common mistake is to omit the factor \((1-h_{ii})^2\) and calculate \(e_i^2h_{ii}/(pMSE)\). For the \(x=2\) case in the second example, this incorrect calculation gives \(4(0.6)/(28/3)=0.2571\), rather than the correct value \(1.6071\). The correction is to divide by \((1-h_{ii})^2=(0.4)^2\), as required by the deletion formula. Omitting this factor understates the effect of deleting a high-leverage observation.

A second frequent error is to set \(p=1\) in simple linear regression because there is one predictor. The fitted model has two parameters—the intercept and the slope—so \(p=2\). Cook’s distance also depends on the units and scale through \(MSE\), which is why comparisons are most meaningful among observations from the same model.

Key takeaway: Cook’s distance combines a case’s residual and leverage to measure the scaled change in all fitted values after that case is deleted. Calculate it using the full-data \(MSE\), the correct parameter count, and the factor \(h_{ii}/(1-h_{ii})^2\).

Check Your Understanding

For each problem, use Cook’s distance with \(p\) counting every fitted coefficient, including an intercept.

  1. A simple regression has \(MSE=9\). For one observation, \(e_i=3\) and \(h_{ii}=0.25\). Calculate \(D_i\). (Answer: \(2/9\approx 0.2222\))
  2. A three-parameter regression has \(MSE=4\), \(e_i=-2\), and \(h_{ii}=0.5\). Calculate Cook’s distance and state whether it is less than 1. (Answer: \(D_i=2/3\approx 0.6667\), which is less than 1)
  3. In a simple regression, \(MSE=5\) and an observation has Cook’s distance \(0.8\). What is the sum of squared changes in all fitted values after deleting it? (Answer: \(8\))
  4. Two observations are in the same simple regression with \(MSE=4\). Case A has \(e=2\), \(h=0.2\); case B has \(e=1\), \(h=0.6\). Calculate both distances and identify which deletion changes the fitted values more. (Answer: \(D_A=0.15625\), \(D_B=0.46875\); case B has the larger Cook’s distance)
  5. A simple regression has \(MSE=3\). An observation has \(h_{ii}=0.4\) and \(D_i=0.5\). Find the absolute value of its residual. (Answer: \(|e_i|=\sqrt{2.7}\approx 1.6432\))

Previous

Next

Back to Tutorials
Clinical Biostats