Why Compare Coefficient Uncertainty?
A case can affect a regression fit in more than one way. It may shift the estimated coefficients, change predictions, or alter how precisely the coefficients are estimated. The covariance ratio focuses on that last question: how does the estimated joint uncertainty of the coefficients change when one observation is removed?
The ratio compares the estimated covariance matrix from the full data with the matrix from a fit that omits one case. It summarizes the change across all coefficients together. It does not say that every coefficient becomes more or less uncertain, and it is not by itself a decision about whether an observation should be removed.
Definition and Conditions
Consider an ordinary least-squares regression with \(n\) observations and \(p\) estimated coefficients, including an intercept if the model has one. Write \(X\) for the \(n\)-by-\(p\) predictor matrix and \(\mathbf{x}_i\) for the column vector corresponding to row \(i\) of \(X\). Suppose \(X\) has full column rank, so the coefficients are estimable. Let \(SSE\) be the full-fit residual sum of squares and \(MSE=SSE/(n-p)\).
For the usual OLS covariance estimate, the full-fit estimated covariance matrix is \(MSE(X'X)^{-1}\). For the fit deleting observation \(i\), use \(X_{-i}\) for the predictor matrix without row \(i\), and \(MSE_{-i}\) for that fit's residual mean square. The deleted-fit covariance estimate is \(MSE_{-i}(X_{-i}'X_{-i})^{-1}\).
The ratio is defined only when both fits have estimable coefficients and positive residual degrees of freedom. In particular, the deleted design matrix must have full column rank, and \(n-p-1\) must be positive.
The determinant summarizes the joint covariance of the coefficient estimates. Because it combines variation and covariation across the coefficient estimates, it is a measure of overall, or generalized, uncertainty rather than the uncertainty of any one coefficient. The ratio is dimensionless: numerator and denominator use the same coefficient units.
A Useful Formula and Its Justification
Let \(h_{ii}=\mathbf{x}_i'(X'X)^{-1}\mathbf{x}_i\) be the leverage of observation \(i\). When the deletion leaves a full-rank design, the covariance ratio can be written using the two residual mean squares and this leverage:
Here \(p\) is the number of coefficients, including the intercept. The formula requires \(h_{ii}<1\), as is the case when deleting the observation leaves the design matrix full rank.
The determinant calculation explains both parts of the formula. Deleting row \(i\) changes the cross-product matrix to \(X_{-i}'X_{-i}=X'X-\mathbf{x}_i\mathbf{x}_i'\). The matrix determinant lemma gives \(\det(X'X-\mathbf{x}_i\mathbf{x}_i')=\det(X'X)(1-h_{ii})\). Taking determinants of the inverse matrices therefore gives a ratio of \(1/(1-h_{ii})\). Separately, multiplying a \(p\)-by-\(p\) matrix by a scalar \(c\) multiplies its determinant by \(c^p\). Applying this to the two \(MSE\) factors gives \((MSE_{-i}/MSE)^p\). Multiplying these two determinant ratios yields the stated result.
The leverage factor alone is greater than one, but the mean-square factor can be smaller or larger than one. The net result can therefore go in either direction; the mean squares and leverage must be considered together.
Interpreting the Ratio
| Value | Interpretation for the deleted fit |
|---|---|
| \(COVRATIO_i>1\) | The determinant of the estimated coefficient covariance matrix is larger: estimated joint uncertainty increases when observation \(i\) is deleted. |
| \(COVRATIO_i<1\) | The determinant is smaller: estimated joint uncertainty decreases when observation \(i\) is deleted. |
| \(COVRATIO_i=1\) | The two covariance determinants are equal, though the covariance matrices themselves need not be identical. |
A value above one does not mean every coefficient has a larger standard error after deletion. One coefficient's uncertainty could rise while another's falls. Likewise, a value below one does not guarantee that each coefficient becomes more precise. The ratio is a compact summary of the joint covariance, so coefficient-specific comparisons require examining the corresponding estimates or standard errors.
Worked Example: Deleting a Central Observation Increases Joint Uncertainty
Consider a hypothetical simple linear regression with an intercept and one predictor. The five data points are \((-2,-3),(-1,-3),(0,4),(1,3),(2,9)\), where the first coordinate is \(x\) and the second is \(y\). The full-fit line is \(\hat{y}=2+3x\). Its residuals, in the order shown, are \(1,-2,2,-2,1\), so \(SSE=1+4+4+4+1=14\). Thus \(p=2\), \(n-p=3\), and \(MSE=14/3\).
Now delete the observation \((0,4)\). The remaining predictor values have mean zero and sum of squares \(10\), so the leverage of the deleted case in the full fit is \(h_{ii}=1/5+0^2/10=0.2\). Refitting to the four remaining points gives \(\hat{y}=1.5+3x\). The residuals are \(1.5,-1.5,-1.5,1.5\), giving deleted-fit \(SSE_{-i}=9\). There are \(4-2=2\) residual degrees of freedom, so \(MSE_{-i}=9/2=4.5\).
Substitute these values into the formula:
The determinant of the estimated coefficient covariance matrix is about 1.162 times as large after deletion. In this hypothetical fit, removing the central observation increases estimated joint coefficient uncertainty by about 16.2%. This is a statement about the covariance determinant, not about every coefficient separately.
Worked Example: Deleting an Endpoint Decreases Joint Uncertainty
For a second hypothetical data set, use the five points \((-2,-1),(-1,-3),(0,1),(1,1),(2,12)\). The full-fit line is \(\hat{y}=2+3x\), with residuals \(3,-2,-1,-4,4\). Their squared values sum to \(46\), so \(MSE=46/3\), since \(n=5\) and \(p=2\).
Delete the endpoint \((2,12)\). Its full-fit leverage is \(1/5+2^2/10=0.6\). The remaining four points have least-squares line \(\hat{y}=x\), with residuals \(1,-2,1,0\). Thus \(SSE_{-i}=1+4+1+0=6\), and \(MSE_{-i}=6/(4-2)=3\).
The covariance ratio is
After deleting this endpoint, the estimated covariance determinant is about 9.57% of its full-fit value. The result says joint uncertainty is much smaller in the deleted fit. It does not establish that the endpoint is erroneous or that deleting it is appropriate; that requires investigating the observation and the analysis context.
Worked Example: Applying the Formula With Three Coefficients
Suppose a hypothetical regression of clinic waiting time on two predictors and an intercept is fitted to \(n=12\) clinics, so \(p=3\). The full fit has \(MSE=25\). A case-deleted refit has \(MSE_{-i}=20\), and the case's full-fit leverage is \(0.30\). Assume both fits have full-rank predictor matrices and positive residual degrees of freedom.
The ratio uses the third power of the mean-square ratio because there are three coefficients:
The deleted fit's covariance determinant is about 73.14% of the full-fit value, so the estimated joint uncertainty is lower after deletion. The mean square falls, while the leverage factor raises the ratio; the combined calculation determines the direction. As in the earlier examples, this summary does not imply that all three coefficient standard errors decrease.
Common Mistakes and Practical Use
- Reversing the ratio. The numerator is the deleted-fit covariance determinant and the denominator is the full-fit determinant. With this convention, a value above one means the determinant increases after deletion.
- Ignoring the number of coefficients. The mean-square ratio is raised to \(p\), not to one. Using the wrong power can materially change the result in a model with several coefficients.
- Reading the ratio as a coefficient-by-coefficient statement. It concerns the determinant of the full covariance matrix. Check individual coefficient uncertainties separately when that is the question.
- Treating a diagnostic value as an instruction to delete. The ratio describes sensitivity of estimated uncertainty to deletion; it does not establish that a case is a data error or should be excluded.
The covariance ratio is most useful alongside other influence diagnostics. It adds a perspective on uncertainty to diagnostics that focus on coefficient changes, fitted values, or residuals. Interpret it as evidence about sensitivity, then verify the observation and consider whether the model and analysis remain appropriate.
Check Your Understanding
Use the definition and formula to answer these questions.
- If a covariance ratio is \(1.4\), what has happened to the deleted-fit covariance determinant relative to the full-fit determinant?
- In a model with four coefficients, why is the mean-square ratio raised to the fourth power?
- A case has leverage \(0.5\), and its deletion leaves the residual mean square unchanged. What is its covariance ratio in a model with two coefficients?
- Can a covariance ratio below one guarantee that every coefficient standard error is smaller after deletion? Explain.
- What conditions on the deleted fit are needed for the covariance ratio to be defined?