Why HC0 Can Understate Variance in Small Samples

Why Residuals Can Mislead About Uncertainty

Heteroskedasticity-robust inference estimates coefficient uncertainty using the fitted regression’s residuals. The difficulty in a small sample is that these residuals are not the unobserved errors themselves: OLS has already used the data to choose coefficients that make residuals small in the directions represented by the regressors. When those residuals stand in for the errors, the resulting HC0 covariance estimate can be too small.

The effect is not a fixed correction that applies equally to every observation or every design. It depends on how many coefficients are fitted, on the leverage of the observations, and, under heteroskedasticity, on how error variances vary across observations. The key finite-sample calculation is the conditional variance of each OLS residual.

Model, Target, and HC0

Condition on a fixed \(n \times k\) design matrix \(X\) of full column rank, where \(k\) counts all fitted coefficients, including an intercept if present. Suppose \(y=X\beta+u\), with \(E(u\mid X)=0\) and \(\operatorname{Var}(u\mid X)=\Omega=\operatorname{diag}(\sigma_1^2,\ldots,\sigma_n^2)\). Thus errors have zero conditional means and are uncorrelated across observations, but their conditional variances may differ. The OLS estimator is \(\hat{\beta}=(X'X)^{-1}X'y\). Write \(B=(X'X)^{-1}\), and let \(e=y-X\hat{\beta}\) be the vector of residuals.

Under these assumptions, the conditional sampling covariance of \(\hat{\beta}\) is \(B X'\Omega X B\). HC0 replaces the unknown observation-specific variances in this expression with squared residuals:

This is an estimator of a covariance matrix, not the covariance itself. Its diagonal entries estimate coefficient variances; their square roots are HC0 standard errors. The question here is whether its conditional expectation equals, falls below, or exceeds the target covariance in a finite sample.

How Fitting Changes Residuals

Define the hat matrix \(H=X(X'X)^{-1}X'\) and residual-maker matrix \(M=I-H\). The fitted values are \(Hy\), and the residuals are \(e=My=Mu\), since \(MX=0\). The matrices \(H\) and \(M\) are symmetric and idempotent; in particular, \(M^2=M\). The diagonal entry \(h_{ii}\) of \(H\) is the leverage of observation \(i\), so \(M_{ii}=1-h_{ii}\).

Because \(e_i=\sum_j M_{ij}u_j\), its conditional variance is a weighted sum of the error variances. Conditional mean zero makes the conditional residual mean zero as well.

This formula matters in two ways. First, a residual is a combination of errors, not simply the error at the same index. Second, its expected square need not estimate \(\sigma_i^2\) exactly, especially when variances differ across observations.

Why HC0 Tends to Be Low Under Homoskedasticity

Under homoskedasticity, the target covariance is \(\sigma^2 B\). Taking the conditional expectation of HC0 and applying the residual-variance result gives

Since \(\operatorname{diag}(1-h_{ii})=I-\operatorname{diag}(h_{ii})\) and \(BX'X B=B\), subtracting the target covariance yields

For any vector \(a\), the quadratic form of \(B X'\operatorname{diag}(h_{ii})X B\) at \(a\) is \(\sum_i h_{ii}(x_i'Ba)^2\), which is nonnegative because \(h_{ii}\geq 0\). Thus the expected HC0 covariance is no greater than the target in any coefficient direction under the stated homoskedastic model. This is a finite-sample result about the conditional expectation, not a claim that each realized HC0 estimate must be below the target.

The design also determines how strong the effect is. With more fitted coefficients relative to the number of observations, residuals have fewer directions in which they can vary. High leverage makes the expected squared residual particularly small for the observations that have that leverage. For a coefficient or contrast, the impact depends on how its design weights align with those observations.

Heteroskedasticity Makes the Direction Less Certain

The homoskedastic result does not automatically extend to unequal error variances. With heteroskedasticity, the exact residual formula is \(\sum_j M_{ij}^2\sigma_j^2\). It can be smaller or larger than the variance \(\sigma_i^2\) of the error at observation \(i\): a residual can inherit variation from other observations through the off-diagonal elements of \(M\).

For example, if a low-leverage observation has a small error variance but its residual combines errors from high-variance observations, its expected squared residual may exceed its own error variance. Conversely, high leverage can suppress the expected squared residual for an observation even when its error variance is large. Consequently, under heteroskedasticity, HC0 is not guaranteed to underestimate the entire covariance matrix in every direction. The general point is that residuals are imperfect proxies in finite samples, and the pattern of variances and the design jointly determine the error.

Common Mistake and Practical Interpretation

A common mistake is to reason, “HC0 is heteroskedasticity-robust, so its residual-based variance estimate is unbiased in a small sample.” Robustness here refers to the covariance formula’s allowance for unequal conditional variances, not exact finite-sample unbiasedness. Under homoskedasticity, the derivation shows a downward expectation; under heteroskedasticity, the direction can depend on the design and variance pattern. A second mistake is to treat the downward tendency as a guarantee for every dataset. The result concerns expectations conditional on \(X\), not the ordering of every realized estimate and target.

Small samples make the distinction consequential because a modest covariance understatement can also make a reported standard error too small and an interval too narrow. The relevant design features are not merely the raw sample size: the number of fitted coefficients, the leverage distribution, and which observations identify a coefficient all matter. HC0 remains a useful large-sample estimator under suitable conditions, but its finite-sample residual shrinkage motivates attention to sample size and design when interpreting robust inference.