Homoskedastic and Heteroskedastic OLS Covariances

Why the OLS covariance depends on the regressor distribution

In random-design regression, the regressors vary across sampled observations. Their distribution therefore affects not only how precisely the design identifies the coefficients, but also how much variability the regression scores carry. The asymptotic covariance of OLS combines these two contributions: a population Gram matrix on either side of a score covariance matrix.

The conditional variance of the error determines whether that middle matrix simplifies. Under conditional homoskedasticity it is a constant multiple of the Gram matrix, yielding the familiar \(\sigma^2 Q^{-1}\). Under conditional heteroskedasticity, observations with different regressor values can contribute different amounts of score variability, and the general sandwich formula is needed.

The vector \(X_i\) may include an intercept. Conditional homoskedasticity means that the conditional error variance is the same for almost every regressor value; conditional heteroskedasticity allows it to vary. Since the conditional mean of \(u_i\) is zero, the conditional variance is \(E[u_i^2\mid X_i]\).

The general asymptotic covariance

Write \(\hat\beta\) for the OLS estimator and define the population covariance of the regression score by \(\Omega=E[X_iX_i^\top u_i^2]\). For the result below, assume \(E[\|X_i\|^2u_i^2]<\infty\), so that \(\Omega\) is finite. The conditional mean restriction gives \(E[X_i u_i]=0\), and the moment condition gives a central limit theorem for the score.

Proof. Let \(Q_n=n^{-1}\sum_{i=1}^n X_iX_i^\top\). When \(Q_n\) is invertible, the OLS normal equations give the exact identity \[ \sqrt{n}(\hat\beta-\beta) = Q_n^{-1}\frac{1}{\sqrt{n}}\sum_{i=1}^n X_i u_i. \] The entries of \(X_iX_i^\top\) have finite expectations, so the law of large numbers gives \(Q_n\xrightarrow{p}Q\). Because \(Q\) is positive definite, \(Q_n^{-1}\xrightarrow{p}Q^{-1}\) on events whose probability tends to one.

Also \(E[X_i u_i]=E[X_iE[u_i\mid X_i]]=0\), and the score \(X_i u_i\) has finite covariance matrix \(\Omega\). The multivariate central limit theorem therefore gives \[ \frac{1}{\sqrt{n}}\sum_{i=1}^n X_i u_i\xrightarrow{d}N(0,\Omega). \] Slutsky's theorem applied to the exact identity yields the stated limit. Finally, iterated expectation gives \[ \Omega=E\!\left[X_iX_i^\top E[u_i^2\mid X_i]\right] =E[X_iX_i^\top\sigma^2(X_i)], \] which proves the equivalent expression. \(\square\)

This is an asymptotic covariance: it is the covariance matrix of the limiting distribution of \(\sqrt n(\hat\beta-\beta)\). It does not assert that the finite-sample covariance of \(\hat\beta\) equals \(n^{-1}Q^{-1}\Omega Q^{-1}\) exactly.

Conditional homoskedasticity: a simplification

Under conditional homoskedasticity, suppose \(\operatorname{Var}(u_i\mid X_i)=\sigma^2\) almost surely, with \(0<\sigma^2<\infty\). The conditional variance no longer weights some regressor values more heavily than others in the score covariance.

Derivation. Since \(\sigma^2\) is constant, it can be taken outside the expectation: \[ \Omega=E[X_iX_i^\top u_i^2] =E[X_iX_i^\top E[u_i^2\mid X_i]] =E[X_iX_i^\top\sigma^2] =\sigma^2Q. \] Substituting into the general formula and using \(Q^{-1}Q=I_p\) gives \[ Q^{-1}\Omega Q^{-1} =Q^{-1}(\sigma^2Q)Q^{-1} =\sigma^2Q^{-1}. \]

Even this simplified expression depends on the regressor distribution through \(Q\). A population design that makes some coefficient directions weakly identified has a Gram matrix that is nearly singular; its inverse then has large entries in those directions. Homoskedasticity simplifies the score variability, but it does not remove the role of random regressors.

Conditional heteroskedasticity: the sandwich formula

When \(\sigma^2(X_i)\) varies, the middle matrix becomes \(E[X_iX_i^\top\sigma^2(X_i)]\), rather than a constant multiple of \(Q\). The outer inverse Gram matrices still describe how score fluctuations translate into coefficient fluctuations. Thus both the distribution of the regressors and the way the error variance changes across regressor values matter.

An alternative way to see the roles of the two matrices is to condition on a realized design. Let \(\mathcal X\) denote the matrix with rows \(X_i^\top\). If \(E[u\mid\mathcal X]=0\) and \(\operatorname{Cov}(u\mid\mathcal X)=D\), then, when \(\mathcal X^\top\mathcal X\) is invertible, \[ \operatorname{Cov}(\hat\beta\mid\mathcal X) =(\mathcal X^\top\mathcal X)^{-1} \mathcal X^\top D\mathcal X (\mathcal X^\top\mathcal X)^{-1}. \] Indeed, \(\hat\beta-\beta=(\mathcal X^\top\mathcal X)^{-1}\mathcal X^\top u\); taking its conditional covariance gives the formula. If \(D=\sigma^2 I_n\), it reduces to \(\sigma^2(\mathcal X^\top\mathcal X)^{-1}\). In random design, dividing the conditional expression by the appropriate sample-size scale leads to the population matrices \(Q\) and \(\Omega\). This conditional calculation is not itself the unconditional asymptotic theorem.

Common mistake and practical interpretation

A common mistake is to assume that the heteroskedastic formula follows from the homoskedastic one by replacing \(\sigma^2\) with the average error variance. That reasoning treats every regressor value as contributing equally to the score variability. In fact, the score is \(X_i u_i\), so its covariance weights the conditional variance by \(X_iX_i^\top\): \[ \Omega=E[X_iX_i^\top\sigma^2(X_i)], \] not \(E[\sigma^2(X_i)]Q\) in general. The replacement is valid only under homoskedasticity or special conditions that make the two matrices coincide.

The random regressor distribution enters twice. First, \(Q=E[X_iX_i^\top]\) captures the population geometry of the design and its information about the coefficients. Second, in the heteroskedastic case, the joint weighting of regressor values and conditional variances determines \(\Omega\). Consequently, two populations with the same average error variance can have different OLS asymptotic covariances if their conditional variance patterns differ across regressor values.