Rates of Convergence for OLS Covariance Estimates

Why a covariance estimate has more than one rate

An OLS covariance estimate combines several random quantities: the observed design matrix, an empirical covariance of regression scores, and often residuals in place of unobserved errors. Each part can converge at a different speed. In particular, consistency only says that an estimation error tends to zero; it does not say that the error is small enough to have a root-\(n\) limit or a useful finite-sample approximation.

We study a fixed-dimensional random-design linear model. The aim is to identify which moment conditions yield consistency and which support root-\(n\) accuracy for the scaled sandwich covariance estimate. The inverse design matrix matters too: even a well-concentrated empirical score covariance cannot compensate for a design matrix that is nearly singular.

The covariance estimate for \(\hat\beta\) itself is \(\hat V_n/n\), not \(\hat V_n\). Thus, if \(\hat V_n-V=O_p(n^{-1/2})\), then \(\hat V_n/n-V/n=O_p(n^{-3/2})\). This describes estimation error relative to the covariance scale \(1/n\); it does not mean that the sampling error of \(\hat\beta\) is \(O_p(n^{-3/2})\).

Rates for the empirical moments

For fixed \(p\), a matrix average is governed by the moment conditions on its entries. The law of large numbers gives consistency when the entries have finite first moments. To obtain a root-\(n\) rate by the ordinary variance argument, the entries need finite second moments.

Here and below, vector norms are Euclidean and matrix norms are Frobenius unless otherwise stated. To see the rate for \(Q_n\), each entry is an average of \(X_{ij}X_{ik}\), and \(E[(X_{ij}X_{ik})^2]\leq E\|X_i\|^4<\infty\). Its variance as an average is therefore \(O(1/n)\), so Chebyshev's inequality gives an \(O_p(n^{-1/2})\) deviation for each entry. There are finitely many entries, so the same order holds for the matrix norm. The argument for \(\Omega_n\) applies to entries \(X_{ij}X_{ik}u_i^2\), whose second moments are bounded by \(E[\|X_i\|^4u_i^4]\).

Residual substitution and the sandwich rate

In general, replacing \(u_i\) by \(\hat u_i\) changes the empirical score covariance. Write \(\delta=\hat\beta-\beta\), so \(\hat u_i=u_i-X_i'\delta\). The following bound uses the Euclidean vector norm and Frobenius matrix norm; specifying them matters because the inequality need not hold with constant one for arbitrary norm choices.

To prove the bound, expand the squared residual and use \(\hat u_i^2-u_i^2=-2u_iX_i'\delta+(X_i'\delta)^2\). The Frobenius norm of \(X_iX_i'\) is \(\|X_i\|^2\), and Cauchy–Schwarz gives \(|X_i'\delta|\leq\|X_i\|\|\delta\|\). Applying the triangle inequality term by term gives

This proves the inequality. \(\square\)

Under the moment conditions in the theorem below, both averages on the right are \(O_p(1)\) by the law of large numbers. For the first, Hölder's inequality gives \(E[\|X_i\|^3|u_i|]\leq(E\|X_i\|^4)^{3/4}(E u_i^4)^{1/4}<\infty\). The second has finite expectation by assumption.

Proof. The score \(X_i u_i\) has mean zero by conditional mean zero and has finite second moment under the assumptions. The multivariate central limit theorem gives \(n^{-1}\sum_iX_i u_i=O_p(n^{-1/2})\). Also \(Q_n-Q=O_p(n^{-1/2})\), and \(Q_n\xrightarrow{p}Q\). Since \(Q\) is positive definite, \(Q_n\) is invertible with probability tending to one and \(Q_n^{-1}=O_p(1)\). The OLS identity

therefore gives \(\delta=O_p(n^{-1/2})\). The residual-substitution bound proves \(\hat\Omega_n-\Omega_n=O_p(n^{-1/2})\); combining it with the empirical-moment rate gives \(\hat\Omega_n-\Omega=O_p(n^{-1/2})\).

The inverse identity \(Q_n^{-1}-Q^{-1}=Q_n^{-1}(Q-Q_n)Q^{-1}\) now gives \(Q_n^{-1}-Q^{-1}=O_p(n^{-1/2})\). Expand the sandwich difference by adding and subtracting one factor at a time:

The empirical matrices and inverse matrices in these products are \(O_p(1)\), while each difference is \(O_p(n^{-1/2})\). Hence every term, and therefore their sum, is \(O_p(n^{-1/2})\). Dividing by \(n\) proves the asserted rate for the covariance estimate itself. \(\square\)

Consistency can hold without a root-n guarantee

The root-\(n\) result uses second moments of the entries being averaged. For \(\Omega_n\), that requires a fourth-moment condition involving both design and error. If only \(E[\|X_i\|^2u_i^2]<\infty\), the law of large numbers can still give \(\Omega_n\xrightarrow{p}\Omega\), but the variance argument used above may no longer apply.

Common mistake: treating consistency as a rate

A common mistake is to reason that, because \(Q_n\to Q\) and \(\Omega_n\to\Omega\), their errors must be of order \(n^{-1/2}\). That does not follow from consistency. The correction is to identify the moments of the entries: finite first moments support the law of large numbers, while finite second moments support the variance calculation for a root-\(n\) bound. Then check the inverse design separately. Positive definiteness of \(Q\) ensures stable inversion asymptotically, but a small minimum eigenvalue in a finite sample can still magnify errors substantially.

Finally, \(V=Q^{-1}\Omega Q^{-1}\) is the asymptotic covariance around \(\beta\) only under exogeneity and the relevant moment conditions. Positive definiteness of \(Q\) alone describes invertibility; it does not establish that OLS is centered at the stated regression parameter.