The Leave-One-Out Identity for OLS

Why a Leave-One-Out Residual Can Be Calculated Directly

An ordinary least-squares (OLS) residual measures the difference between an observed response and the fitted value from a model that used that observation in fitting. A leave-one-out residual asks a different question: how far is the observed response from the prediction made by a model fitted without that case? The second quantity is useful when examining how a case fares under a fit that did not use its response to estimate the coefficients.

Refitting the model once for every observation is possible, but it is unnecessary for OLS when the required rank condition holds. The hat matrix provides a direct identity: the leave-one-out residual is the ordinary residual divided by one minus the case’s hat value. This tutorial derives the identity and works through its use.

Model, Hat Values, and Deleted Residuals

Let \(\mathbf{y}\) be the \(n\)-vector of observed responses and let \(\mathbf{X}\) be the \(n \times p\) design matrix, including a column of ones if the model has an intercept. The \(i\)th row of \(\mathbf{X}\) is \(\mathbf{x}_i^\mathsf{T}\), where \(\mathbf{x}_i\) is a \(p\)-vector. The ordinary linear model notation is \(\mathbf{y}=\mathbf{X}\boldsymbol{\beta}+\boldsymbol{\varepsilon}\). For the algebra below, the essential conditions are that OLS is used and \(\mathbf{X}\) has full column rank. No normality or constant-variance assumption is needed for the identity itself.

The full-data OLS estimate is \(\hat{\boldsymbol{\beta}}=(\mathbf{X}^\mathsf{T}\mathbf{X})^{-1}\mathbf{X}^\mathsf{T}\mathbf{y}\). The fitted response vector is \(\hat{\mathbf{y}}=\mathbf{H}\mathbf{y}\), where \(\mathbf{H}=\mathbf{X}(\mathbf{X}^\mathsf{T}\mathbf{X})^{-1}\mathbf{X}^\mathsf{T}\) is the hat matrix. Its \(i\)th diagonal element, \(h_{ii}\), is the hat value for case \(i\). The ordinary residual is \(e_i=y_i-\hat{y}_i\).

For this deleted fit to have a unique OLS coefficient estimate, the design matrix with row \(i\) removed must still have full column rank. Under that condition, the leave-one-out identity is:

This is an exact algebraic result for the observed data. It is not an approximation, and it does not require a probability model for the errors.

Deriving the Identity

The derivation uses the OLS normal equations: the vector of OLS residuals is orthogonal to every column of the design matrix. Let \(\mathbf{X}_{-i}\) and \(\mathbf{y}_{-i}\) denote the design matrix and response vector after case \(i\) is removed. Let \(\mathbf{e}_{-i}\) be the vector of residuals for those cases under the full-data fit. Define the coefficient difference as \(\boldsymbol{\delta}=\hat{\boldsymbol{\beta}}-\hat{\boldsymbol{\beta}}_{(i)}\).

First, apply the full-data normal equations, \(\mathbf{X}^\mathsf{T}\mathbf{e}=\mathbf{0}\), and separate case \(i\)'s contribution. Then apply the deleted-fit normal equations. Together these give:

The second equation holds because the residuals for the remaining cases under the deleted fit are \(\mathbf{y}_{-i}-\mathbf{X}_{-i}\hat{\boldsymbol{\beta}}_{(i)}=\mathbf{e}_{-i}+\mathbf{X}_{-i}\boldsymbol{\delta}\). Substituting the first equation into the second yields:

The deleted residual for case \(i\) is its response minus its deleted-fit prediction. Since \(\hat{\boldsymbol{\beta}}=\hat{\boldsymbol{\beta}}_{(i)}+\boldsymbol{\delta}\), it follows that:

To connect this expression to \(h_{ii}\), set \(\mathbf{A}=\mathbf{X}_{-i}^\mathsf{T}\mathbf{X}_{-i}\) and \(q=\mathbf{x}_i^\mathsf{T}\mathbf{A}^{-1}\mathbf{x}_i\). Because deleting the row leaves full column rank, \(\mathbf{A}\) is invertible. Also, \(\mathbf{X}^\mathsf{T}\mathbf{X}=\mathbf{A}+\mathbf{x}_i\mathbf{x}_i^\mathsf{T}\). The rank-one inverse identity gives:

Multiplying on the left and right by \(\mathbf{x}_i^\mathsf{T}\) and \(\mathbf{x}_i\), respectively, shows that the full-data hat value is \(h_{ii}=q-q^2/(1+q)=q/(1+q)\). Therefore \(q=h_{ii}/(1-h_{ii})\), and \(1+q=1/(1-h_{ii})\). Substituting into the expression for \(d_i\) proves the identity. Since \(q\) is finite and nonnegative, \(h_{ii}=q/(1+q)<1\).

Using the Identity

The calculation requires only the full-data residual and hat value for the case of interest. The sign of \(d_i\) matches the sign of \(e_i\), because \(1-h_{ii}\) is positive when the rank condition holds. Its magnitude is at least as large as the ordinary residual’s magnitude, since \(0<1-h_{ii}\leq 1\). The difference can be small when the hat value is small and more substantial when the hat value is close to one.

Conditions and a Common Mistake

The rank condition matters. If deleting case \(i\) makes the remaining design matrix rank deficient, then the deleted OLS coefficients are not uniquely determined by the usual full-rank formula. In that situation, the derivation using \((\mathbf{X}_{-i}^\mathsf{T}\mathbf{X}_{-i})^{-1}\) does not apply; correspondingly, \(h_{ii}=1\), so the identity would have a zero denominator. Do not calculate \(e_i/(1-h_{ii})\) in that case and treat it as a finite deleted residual.

• Wrong reasoning: “The ordinary residual is the leave-one-out residual because both compare the observed response with a regression prediction.”
• Correction: The ordinary residual uses a fit that included the case. The leave-one-out residual uses a fit that excluded it, so the two are generally different. Under the stated rank condition, the exact conversion is \(d_i=e_i/(1-h_{ii})\).
• Another common error: Dividing by \(h_{ii}\) instead of \(1-h_{ii}\). The adjustment factor is the reciprocal of one minus the hat value.

The identity makes the relationship between residual size and leverage explicit: for the same ordinary residual, a larger hat value produces a larger absolute deleted residual. It is important to keep the quantities distinct, however. The identity calculates a leave-one-out residual; it does not by itself provide a standardized residual, a significance test, or a causal interpretation.