Influence in Multiple Regression

Why Predictor Combinations Matter

In multiple regression, a case can be unusual in the joint arrangement of its predictor values without being extreme on any predictor considered separately. For example, if two predictors usually increase together, a case with an ordinary value of each predictor but a combination that breaks that pattern may be far from the rest of the data in predictor space. Such a case can have high leverage and may strongly affect the fitted model.

This is a multivariable feature of regression diagnostics. Looking at one predictor at a time can miss it. Leverage describes the case’s position in the design matrix; influence describes how much the fitted regression changes because of the case. A high value of one diagnostic is a reason to examine a case, not proof that the observation is erroneous or that it must be removed.

Leverage in Multiple Regression

Consider an ordinary least squares model with $k$ predictors and an intercept. Let $X$ be the design matrix, with one row per case and one column for each model parameter, including the intercept. Assume $X$ has full column rank, so the regression coefficients are identifiable. The hat matrix is $H=X(X^\mathsf{T}X)^{-1}X^\mathsf{T}$, and the leverage of case $i$ is its diagonal entry $h_{ii}$. We write this entry as $h_i$.

Definition: The leverage $h_i$ is the $i$th diagonal entry of the hat matrix. It depends on the predictor values and the design matrix, not on the response values. Leverages satisfy $0 \le h_i \le 1$, and their sum equals the number of estimated regression parameters, including the intercept.

In a multiple regression, $h_i$ depends on how the full vector of predictor values compares with the other predictor vectors. It is not enough to check whether each predictor value falls inside its separate observed range. When predictors are strongly related, a combination that departs from their usual relationship can be unusual even if every component value looks ordinary.

A commonly used screening guide is to flag leverage greater than $2p/n$, where $p$ is the number of fitted parameters, including the intercept. This is a heuristic, not a formal test or a universal boundary. The average leverage is $p/n$, so the guide identifies cases noticeably above that average; it does not establish that a case is influential or problematic.

A Multiple-Regression Example

Worked Example: Finding Leverage for an Unusual Combination

Suppose a hypothetical data set has two predictors, $x$ and $z$, and nine cases. The values of $x$ are $-3,-2,-1,0,1,2,3,0,0$. For most cases, $z=x$; the three cases with $x=0$ have $z$ values $-0.5,1,$ and $-0.5$. Focus on case 8, with $x=0$ and $z=1$. Both values are within the observed range of their respective predictors, $-3$ to $3$. Yet this pair departs from the main pattern $z=x$.

To calculate its leverage, define $q=z-x$. The predictors can equivalently be represented by $x$ and $q$, since $z=x+q$. This change of predictor columns leaves the column space of the design matrix—and hence its hat matrix—unchanged. Across the nine cases, $x$ has mean zero and sum of squares $28$; $q$ is zero except for values $-0.5,1,-0.5$ in cases 4, 8, and 9. Thus $q$ also has mean zero, is orthogonal to $x$, and has sum of squares $1.5$.

With the intercept, $x$, and $q$ as mutually orthogonal design columns, the leverage for a case with values $x_i$ and $q_i$ is the sum of its contributions from the three columns:

h_i=\frac{1}{9}+\frac{x_i^2}{28}+\frac{q_i^2}{1.5}.

For case 8, $x_8=0$ and $q_8=1$, so

h_8=\frac{1}{9}+\frac{0^2}{28}+\frac{1^2}{1.5} =\frac{1}{9}+\frac{2}{3} =\frac{7}{9}\approx 0.7778.

There are three fitted parameters, so the average leverage is $3/9=0.3333$, and the heuristic $2p/n$ guide is $6/9=0.6667$. Case 8 exceeds that guide. Its leverage is high because it departs from the joint predictor pattern, not because either predictor value is individually extreme.

From Leverage to Influence

Leverage alone does not say how much a case affects the fitted regression. A case can have high leverage but a small residual, in which case its fitted value may be close to what the model predicts and its influence may be limited. Conversely, a large residual at low leverage does not necessarily produce a large change to the overall fit. Influence measures combine information about the predictor position and the response discrepancy.

For the calculations below, assume the usual multiple linear regression model: the conditional mean of the response is linear in the included predictors, errors are independent with mean zero and constant variance, and the design matrix has full column rank. Normal errors are additionally needed for exact normal- and $t$-based distributional results, but no such probability calculation is required for the diagnostic arithmetic here. The examples are hypothetical; the calculations do not by themselves validate the model assumptions.

Definition: Let $e_i=y_i-\hat{y}_i$ be the ordinary residual for case $i$, let $MSE=SSE/(n-p)$, and let $s=\sqrt{MSE}$. Cook’s distance is $$ D_i=\frac{e_i^2}{p\,MSE}\frac{h_i}{(1-h_i)^2}. $$ It summarizes the change in the fitted regression associated with deleting a case, scaled by the number of fitted parameters and the model’s residual variation.

DFFITS focuses on the change in the fitted value for the case itself. Define the deleted residual variance estimate $s_{(i)}^2$ by fitting the model without case $i$, with $n-p-1$ residual degrees of freedom. The externally studentized deleted residual is $t_i=e_i/[s_{(i)}\sqrt{1-h_i}]$, and $DFFITS_i=t_i\sqrt{h_i/(1-h_i)}$. A large absolute DFFITS indicates a large standardized change in the fitted value for that case. Its sign indicates the direction of that change under this convention.

Calculating Influence Measures

Worked Example: Computing Cook’s Distance and DFFITS

Continue with the same nine hypothetical cases, now with responses shown below. Fitting a model with an intercept and predictors $x$ and $z$ gives coefficients $10,2,$ and $3$, respectively. The ordinary residuals are included so the diagnostic calculations can be checked directly.

Case	$x$	$z$	$y$	Residual $e_i$
1	-3	-3	-5	0
2	-2	-2	-1	-1
3	-1	-1	4.5	-0.5
4	0	-0.5	9.5	1
5	1	1	14.5	-0.5
6	2	2	19	-1
7	3	3	25	0
8	0	1	14	1
9	0	-0.5	9.5	1

For example, the fitted response for case 8 is $10+2(0)+3(1)=13$, so its residual is $14-13=1$. Squaring and summing all nine residuals gives $SSE=5.5$. There are $p=3$ fitted parameters and $n-p=6$ residual degrees of freedom, so

MSE=\frac{5.5}{6}=\frac{11}{12}\approx 0.9167.

For case 8, $e_8=1$ and $h_8=7/9$. Substitution in Cook’s distance gives

D_8=\frac{1^2}{3(11/12)} \frac{7/9}{(1-7/9)^2} =\frac{4}{11}\frac{63}{4} =\frac{63}{11}\approx 5.727.

This is a large Cook’s distance by common screening guides, so case 8 merits attention. It is not a reason on its own to delete the case: the size of the diagnostic neither identifies the cause nor establishes that the observation is invalid.

For DFFITS, first find the residual sum of squares and variance estimate after deleting case 8. The leave-one-out identity gives $SSE_{(8)}=SSE-e_8^2/(1-h_8)$, hence

SSE_{(8)}=5.5-\frac{1}{2/9}=1, \qquad s_{(8)}^2=\frac{1}{9-3-1}=\frac{1}{5}=0.2000.

The externally studentized deleted residual and DFFITS are therefore

t_8=\frac{1}{\sqrt{0.2000}\sqrt{2/9}} =\sqrt{22.5}\approx 4.743, \qquad DFFITS_8=\sqrt{22.5}\sqrt{\frac{7/9}{2/9}} =\sqrt{\frac{315}{4}}\approx 8.874.

Thus deleting case 8 changes its own fitted value substantially relative to the deleted-fit standard error. DFFITS is dimensionless, as is Cook’s distance; neither value is a change measured in the original response units.

A Short Justification: Why the Leverage Decomposition Works

The leverage calculation above follows directly from the geometry of orthogonal design columns. If the columns of $X$ are mutually orthogonal, then $X^\mathsf{T}X$ is diagonal. The hat matrix is the sum of the projection matrices for the individual columns. For a column $v$, that column’s projection matrix is $vv^\mathsf{T}/(v^\mathsf{T}v)$, whose $i$th diagonal entry is $v_i^2/(v^\mathsf{T}v)$. Summing these contributions for the intercept, $x$, and $q$ gives

h_i=\frac{1^2}{9}+\frac{x_i^2}{\sum_j x_j^2} +\frac{q_i^2}{\sum_j q_j^2} =\frac{1}{9}+\frac{x_i^2}{28}+\frac{q_i^2}{1.5}.

For the original predictors $x$ and $z$, the design columns are not orthogonal. But replacing $z$ by $q=z-x$ is an invertible change of predictor columns: the two designs have the same column space. Since least squares projection onto a column space is unchanged by a different basis for that space, this calculation gives the leverage from the original multiple-regression model.

Comparing Diagnostics and Avoiding Overinterpretation

Worked Example: A High-Leverage Case Is Not Automatically Influential

In the same data set, case 2 has $x=-2$, $q=0$, and residual $e_2=-1$. Its leverage is

h_2=\frac{1}{9}+\frac{(-2)^2}{28}+\frac{0^2}{1.5} =\frac{1}{9}+\frac{1}{7} =\frac{16}{63}\approx 0.2540.

This is below the $2p/n=0.6667$ screening guide. Its Cook’s distance, using the same $MSE=11/12$, is

D_2=\frac{(-1)^2}{3(11/12)} \frac{16/63}{(1-16/63)^2} \approx 0.1659.

Case 2 has a nonzero residual but much less leverage and a much smaller Cook’s distance than case 8. The contrast illustrates why influence assessment combines information: a residual alone does not describe how unusual the predictor combination is, and leverage alone does not describe how far the response departs from the fitted value.

A case such as case 8 can affect several fitted coefficients at once because the regression must accommodate an unusual predictor combination. Cook’s distance summarizes overall change across the fitted values, while DFFITS emphasizes the fitted value at the case itself. Coefficient-specific diagnostics answer a different question. It is useful to consult multiple diagnostics, but the measures are related and should not be treated as independent votes.

The screening threshold $2p/n$ helps identify leverage that is large relative to the design’s average. Other rules of thumb are sometimes used for Cook’s distance and DFFITS, but none is a universal decision boundary. The model size, sample size, predictor structure, and consequences of the analysis all matter. A case just below a guide is not necessarily harmless, and one above it is not automatically a mistake.

Common Mistake

Common mistake: “Case 8 is not an outlier because both $x=0$ and $z=1$ are inside their separate observed ranges, so it cannot have high leverage.” This reasoning checks each predictor in isolation. Leverage depends on the joint predictor pattern. Here $q=z-x=1$ is unusual relative to the other cases, producing $h_8=7/9$. The correction is to assess the case in the full design matrix and then consider residual-based influence measures as well.

High leverage and influence are diagnostic descriptions, not labels for bad data. Before changing a model or excluding a case, consider whether the values were recorded correctly, whether the case belongs to the intended population, and whether the model adequately represents the relationship. A real case can be both valid and influential. Removing it without a defensible reason can hide meaningful structure or change the question the analysis answers.

Key takeaway: In multiple regression, unusual combinations of ordinary predictor values can create high leverage. Calculate leverage from the full design, then use measures such as Cook’s distance and DFFITS to assess influence. Treat flags as prompts for careful investigation, not automatic instructions to remove observations.

Check Your Understanding

Use the definitions and calculations in this tutorial to answer the following questions.

Why can a case have high leverage even when none of its predictor values is individually outside the observed range?
For case 8, identify the quantity that makes its predictor combination unusual and explain how it contributes to $h_8$.
Why does a high leverage value by itself not establish that a case strongly affects the fitted regression?
In the worked example, what does the positive DFFITS value indicate, and why is it dimensionless?
Case 8 exceeds a common leverage screening guide. What can this flag reasonably prompt, and what conclusion does it not justify by itself?