Robust Inference Under Conditional-Mean Misspecification

OLS When the Conditional Mean Is Not Linear

A linear regression can be useful even when the relationship between the outcome and regressors is not a straight line. In that setting, OLS does not generally estimate a coefficient describing a correctly specified conditional mean. It estimates the coefficients of a particular best linear approximation, defined by the population distribution of the data. Inference can be valid for that approximation, but it must be interpreted as inference about the projection—not automatically about a causal effect or a structural parameter.

This distinction matters because a regression can have a well-defined population target even when its conditional-mean model is wrong. Heteroskedasticity-robust covariance estimation addresses uncertainty around that target under suitable sampling and moment conditions. It does not repair the functional form, change the target, or turn an association into a causal effect.

The Best Linear Projection

Let $Y$ be a scalar outcome and let $Z$ be a $k$-dimensional regressor vector, including a constant if the regression has an intercept. The observations $(Y_i,Z_i)$, for $i=1,\ldots,n$, are assumed to be independent and identically distributed draws from a population. No assumption that $E[Y\mid Z]$ is linear is made.

Definition: If $E[Y^2]<\infty$, $Q=E[ZZ']$ is finite and positive definite, the best linear projection coefficient is $$ \beta^*=\arg\min_b E[(Y-Z'b)^2]=Q^{-1}E[ZY]. $$ The projection residual is $u^*=Y-Z'\beta^*$.

The moment condition $E[Y^2]<\infty$ is important: it ensures the squared-error objective is finite for finite $b$. Together with $E[ZZ']$ finite, it also ensures $E[ZY]$ exists, by the Cauchy–Schwarz inequality applied to the components of $Z$ and $Y$. Positive definiteness of $Q$ makes the minimizer unique.

Differentiating the objective with respect to $b$ gives the projection normal equations $E[Z(Y-Z'\beta^*)]=0$. Thus $E[Zu^*]=0$: the projection residual is orthogonal, on average, to each included regressor. This is not the same as saying $E[u^*\mid Z]=0$. The latter would imply that $Z'\beta^*$ equals the conditional mean almost surely, which need not be true.

Write $m(Z)=E[Y\mid Z]$, when this conditional expectation exists. Then $$ u^*=\{m(Z)-Z'\beta^*\}+\{Y-m(Z)\}. $$ The first term is the conditional-mean approximation error; it can vary systematically with $Z$. The second is the outcome's deviation from its conditional mean. Even though $E[Zu^*]=0$, the approximation error may be nonzero at particular regressor values.

Worked Example: Projecting a Curved Mean onto a Line

Suppose, hypothetically, that $X$ takes the values $-1,0,1$, each with probability $1/3$, and $Y=X^2+\varepsilon$, where $E[\varepsilon\mid X]=0$ and $\operatorname{Var}(\varepsilon\mid X)=0.25$. Fit a linear projection with $Z=(1,X)'$.

The population moments are $E[X]=0$, $E[X^2]=2/3$, and $E[Y]=E[X^2]=2/3$. Also $E[XY]=E[X^3]+E[X\varepsilon]=0$, by symmetry and the conditional mean-zero assumption. Therefore $$ Q=\begin{pmatrix}1&0\\0&2/3\end{pmatrix}, \qquad E[ZY]=\begin{pmatrix}2/3\\0\end{pmatrix}, \qquad \beta^*=\begin{pmatrix}2/3\\0\end{pmatrix}. $$

The projection line is therefore $2/3+0X$, although the conditional mean is the curve $m(X)=X^2$. The projection residual has conditional means $1/3$ at $X=-1$, $-2/3$ at $X=0$, and $1/3$ at $X=1$. These are not all zero, but their regressor-weighted averages satisfy the normal equations.

Why the Robust Covariance Is a Sandwich

Let $\widehat{\beta}$ be the OLS estimator, and define the fitted residual $\widehat{u}_i=Y_i-Z_i'\widehat{\beta}$. Under the iid sampling assumption, a finite nonsingular $Q$, and suitable moment conditions such as $E[\lVert Z u^*\rVert^2]<\infty$, the projection normal equations give the large-sample distribution of OLS. The key is to center the estimator on $\beta^*$, not on coefficients from a possibly false conditional-mean model.

Theorem: Under the stated conditions, with a central limit theorem for $Z u^*$ and the usual consistency conditions for sample moments, $$ \sqrt{n}(\widehat{\beta}-\beta^*)\ \xrightarrow{d}\ N(0,Q^{-1}\Omega Q^{-1}), \qquad \Omega=E[ZZ'u^{*2}]. $$ The corresponding heteroskedasticity-consistent HC0 covariance estimator is $$ \widehat{V}_{HC0}=(Z'Z)^{-1} \left(\sum_{i=1}^n Z_iZ_i'\widehat{u}_i^{\,2}\right) (Z'Z)^{-1}. $$

Here $Z$ in the estimator denotes the matrix whose $i$th row is $Z_i'$. The middle matrix estimates the variability of the observation-level terms $Z_i u_i^*$; the two outside matrices account for the curvature of the least-squares objective. The formula is called a sandwich because of this outside–middle–outside structure. For consistency of the sample meat, integrability conditions are needed; for example, sufficiently high finite moments of $Y$ and $Z$ can ensure the required laws of large numbers.

Derivation of the Projection Covariance

The finite-sample OLS normal equations imply the following exact identity whenever $Z'Z$ is invertible:

\widehat{\beta}-\beta^* =(Z'Z)^{-1}\sum_{i=1}^n Z_i u_i^* =\left(\frac{1}{n}\sum_{i=1}^n Z_iZ_i'\right)^{-1} \left(\frac{1}{n}\sum_{i=1}^n Z_i u_i^*\right).

The law of large numbers makes the first matrix in parentheses converge to $Q$. Since $E[Zu^*]=0$, the central limit theorem makes the second term, after multiplication by $\sqrt n$, converge to a mean-zero normal vector with covariance $\Omega$. Applying the continuous mapping theorem to the matrix inverse and then the standard linear transformation rule for covariance gives

\sqrt{n}(\widehat{\beta}-\beta^*)\ \xrightarrow{d}\ N\!\left(0,Q^{-1}\Omega(Q^{-1})'\right).

Because $Q=E[ZZ']$ is symmetric, $Q^{-1}=(Q^{-1})'$, producing the stated sandwich covariance. The sample residuals replace the unobserved $u_i^*$ in HC0. Under the consistency conditions, this substitution and the sample-moment replacements estimate the asymptotic covariance. This is a large-sample result; HC0 is not guaranteed to give exact finite-sample coverage.

Worked Example: Robust Uncertainty for the Curved-Mean Projection

Continue with the hypothetical distribution in the preceding example. For the slope component, the relevant diagonal element of $\Omega$ is $E[X^2u^{*2}]$. At $X=-1$ and $X=1$, the conditional mean of $u^*$ is $1/3$ and its conditional variance is $0.25$, so $E[u^{*2}\mid X]=1/9+1/4=13/36$. At $X=0$, $X^2=0$, so that value contributes nothing to the slope element. Hence $$ \Omega_{22}=E[X^2u^{*2}] =\frac{2}{3}\left(\frac{1}{9}+\frac{1}{4}\right) =\frac{13}{54}. $$

Since $Q_{22}=E[X^2]=2/3$, the asymptotic variance of the slope estimator, multiplied by $n$, is $$ (Q^{-1}\Omega Q^{-1})_{22} =\frac{13/54}{(2/3)^2} =\frac{13}{24} \approx 0.5417. $$

For a hypothetical large sample of $n=300$ observations from this distribution, the resulting asymptotic standard error is $\sqrt{(13/24)/300}\approx0.04249$. If the estimated slope were zero and this population-based standard error were used as an idealized illustration, a normal-approximation 95% interval would be $0\pm1.96(0.04249)$, or approximately $[-0.0833,0.0833]$ per unit of $X$. This concerns the linear projection slope, whose population value is zero. In an actual analysis, the interval would use the sample HC0 estimate, and its large-sample interpretation would still concern the projection parameter.

What Robust Inference Does—and Does Not—Protect

A heteroskedasticity-robust standard error allows the conditional variance of the projection residual to vary with the regressors. Under the assumptions supporting the sandwich limit, robust $t$- or Wald-based procedures can provide asymptotically valid inference about $\beta^*$, or about specified linear combinations of its components. They do not require the conditional mean to be linear. But robust covariance estimation does not make a linear approximation exact, remove omitted-variable bias relative to a causal target, or guarantee good small-sample performance.

The next calculation illustrates why the sandwich and the conventional homoskedastic formula can differ when both conditional-mean misspecification and heteroskedasticity are present.

Worked Example: Comparing Robust and Conventional Variance Targets

Suppose $X$ again takes $-1,0,1$ with equal probabilities and $Y=X^2+\varepsilon$, so the projection residual's conditional mean is $1/3,-2/3,1/3$ at those values. Now let the hypothetical conditional variances of $\varepsilon$ be $1$ at $X=-1,1$ and $0.25$ at $X=0$, with conditional mean zero at each value. The projection coefficient remains $(2/3,0)'$.

For the slope, the robust asymptotic variance multiplied by $n$ is $$ \frac{E[X^2u^{*2}]}{(E[X^2])^2} =\frac{(2/3)(1/9+1)}{(2/3)^2} =\frac{20/27}{4/9} =\frac{5}{3} \approx1.667. $$

The conventional homoskedastic formula instead uses $E[u^{*2}]/E[X^2]$. Its residual second moment is $$ E[u^{*2}] =\frac{2}{3}\left(\frac{1}{9}+1\right) +\frac{1}{3}\left(\frac{4}{9}+\frac{1}{4}\right) =\frac{20}{27}+\frac{25}{108} =\frac{35}{36}. $$ Thus its variance target multiplied by $n$ is $(35/36)/(2/3)=35/24\approx1.458$, not $5/3$. For $n=400$, the corresponding idealized standard errors are $\sqrt{(5/3)/400}\approx0.06455$ robustly and $\sqrt{(35/24)/400}\approx0.06038$ under the conventional formula. The difference arises because the conditional variance is not constant and the projection residual includes systematic approximation error. These calculations compare asymptotic targets; they are not a claim about exact standard errors in every finite sample.

Projection Inference Is Not Causal Inference

A projection coefficient is defined by the joint population distribution of outcomes and regressors. A causal effect instead concerns what would happen to outcomes under an intervention, and its identification requires assumptions that connect observed data to those intervention outcomes. Correctly estimating a projection coefficient—even with a consistent robust covariance estimator—does not supply those assumptions.

Worked Example: A Precisely Estimated Association Can Be Confounded

Suppose, hypothetically, an outcome satisfies $Y=2+3D+4U$, where $D$ is a binary treatment indicator and $U$ is a binary background characteristic. Assume the treatment is associated with $U$: $P(U=1\mid D=1)=0.8$ and $P(U=1\mid D=0)=0.2$. Regress $Y$ on an intercept and $D$, omitting $U$.

Because $D$ is binary, the population OLS slope is the difference in conditional means. Those means are $$ E[Y\mid D=1]=2+3+4(0.8)=8.2, \qquad E[Y\mid D=0]=2+4(0.2)=2.8, $$ so the projection slope is $8.2-2.8=5.4$ outcome units per treatment-indicator unit.

Under the stated outcome equation, changing $D$ while holding $U$ fixed changes $Y$ by $3$ units, not $5.4$. The extra $2.4$ in the observed group difference reflects the groups' different distributions of $U$. Robust inference could quantify uncertainty about the projection slope of $5.4$; it would not make that slope an estimate of the causal effect $3$.

Common Mistakes and Practical Interpretation

Claiming $E[u^*\mid Z]=0$ because OLS residuals are orthogonal to regressors. The projection only guarantees $E[Zu^*]=0$. A nonlinear conditional mean can leave systematic conditional residual means.
Treating robust standard errors as a correction for misspecification. They adjust the estimated covariance for heteroskedasticity; they do not change the projection target or make the fitted line equal the conditional mean.
Calling a projection coefficient causal because its robust interval excludes zero. The interval supports a statement about the population projection under the sampling assumptions. Causal language requires a defensible identification strategy.
Using the sandwich formula without moment conditions. Finite, nonsingular $E[ZZ']$ alone is insufficient to justify the squared-error objective or $E[ZY]$. Conditions such as $E[Y^2]<\infty$ are also needed.
Interpreting a large-sample interval as exact in a small sample. The robust covariance result is asymptotic. Leverage, sample size, and other finite-sample features can still affect its accuracy.

A careful report names the target explicitly: for example, “the coefficient is the population linear-projection slope of outcome on these regressors,” followed by the robust uncertainty estimate and its units. If the conditional mean is thought to be nonlinear, the line can still summarize a chosen population relationship, but its meaning depends on the distribution of the regressors and the linear specification.

Check Your Understanding

Answer each question by distinguishing the projection target from assumptions about the conditional mean or causal effects.

What moment condition, in addition to finite nonsingular $E[ZZ']$, helps ensure that the squared-error projection objective is finite?
Why does $E[Zu^*]=0$ not imply $E[u^*\mid Z]=0$?
In the sandwich covariance, what population quantity is represented by $\Omega=E[ZZ'u^{*2}]$?
Does HC0 change the population coefficient that OLS targets when the conditional mean is nonlinear? Explain.
In the confounding example, what does robust inference quantify, and why does it not establish the causal effect?