Robust Regression Versus Robust Standard Errors

Two Different Meanings of “Robust”

In regression, the word robust can describe two different changes to an analysis. A robust regression method changes how the coefficients are fitted, often by reducing the influence of large residuals. A robust standard-error method keeps the ordinary least-squares coefficients but changes how their uncertainty is estimated. These approaches address different concerns, so one is not a substitute for the other.

This distinction matters in practice. If an unusual response value has pulled the fitted line toward itself, changing the standard-error calculation does not move that line back. If the fitted coefficients are reasonable but the error variance may differ across observations, heteroskedasticity-robust standard errors can provide a different uncertainty estimate without changing the coefficients.

The OLS Fit and Its Usual Uncertainty Estimate

Write the linear regression model for \(n\) observations as \(Y=X\beta+\varepsilon\). Here \(Y\) is the vector of responses, \(X\) is the design matrix containing the predictor values (and usually a column of ones for the intercept), \(\beta\) is the vector of coefficients, and \(\varepsilon\) is the vector of errors. Let \(q\) be the number of coefficients, including the intercept when present. Assume \(X\) has full column rank, so the coefficients can be estimated uniquely.

For the standard OLS coefficient and covariance results, assume that, conditional on the observed predictors, the errors have mean zero and observations are independent. The usual OLS covariance formula additionally assumes a common conditional error variance, \(\operatorname{Var}(\varepsilon_i\mid X)=\sigma^2\). The zero conditional mean assumption says the regression model captures the conditional mean; it does not say that every observed response lies exactly on that mean.

The residual \(e_i=y_i-x_i^\mathsf{T}\hat{\beta}_{\mathrm{OLS}}\) is the observed response minus its fitted value. Under the usual assumptions, the estimated standard error for a coefficient is the square root of the corresponding diagonal entry of the estimated covariance matrix. If the error variance changes across observations, the usual formula can give an inaccurate estimate of coefficient uncertainty even when the conditional mean model is appropriate.

What Heteroskedasticity-Robust Standard Errors Change

Suppose errors remain independent and have conditional mean zero, but their variances may differ: \(\operatorname{Var}(\varepsilon_i\mid X)=\sigma_i^2\). The OLS coefficient formula is still the same. Its conditional covariance is now a “sandwich” rather than the usual common-variance formula. The HC0 estimate replaces the unknown error variances with squared OLS residuals.

Here \(x_i\) is the vector of predictors for observation \(i\), including the intercept entry if there is one, and \(\Omega\) is the diagonal matrix of the conditional error variances. The HC0 standard error is the square root of a diagonal element of the HC0 covariance estimate. Some software reports a finite-sample adjustment, such as HC1, which multiplies this covariance estimate by \(n/(n-q)\). Different adjustments can give different standard errors; they do not change the OLS coefficients.

Why the Sandwich Formula Appears

A short derivation makes clear why a variance correction leaves the point estimate alone. Under the linear model, subtracting \(\beta\) from the OLS formula gives an exact expression for its estimation error:

Conditional on \(X\), the matrix multiplying \(\varepsilon\) is fixed. For any fixed matrix \(A\) and random vector \(Z\), \(\operatorname{Cov}(AZ)=A\operatorname{Cov}(Z)A^\mathsf{T}\). Applying this identity with \(A=(X^\mathsf{T}X)^{-1}X^\mathsf{T}\), and using \(\operatorname{Cov}(\varepsilon\mid X)=\Omega\), gives the sandwich covariance formula. If \(\Omega=\sigma^2 I\), it reduces to \(\sigma^2(X^\mathsf{T}X)^{-1}\), the familiar common-variance result. Estimating each variance contribution with \(e_i^2\) gives HC0. This changes the estimate of covariance, not the equation that defined \(\hat{\beta}_{\mathrm{OLS}}\).

What Robust Regression Changes

Robust regression changes the coefficient estimate by changing the loss function being minimized. Least squares minimizes the sum of squared residuals, so a residual twice as large contributes four times as much loss. A robust loss can reduce the growth of a residual’s contribution once it is large. For example, Huber loss is quadratic for residuals near zero and grows linearly for sufficiently large residuals.

Here \(s\) is a residual scale and \(c\) is a chosen positive cutoff. The loss and scale choice define a different fitting procedure from OLS. A robust loss may limit the effect of a large residual, but it does not automatically provide a valid standard error: uncertainty must be estimated for the chosen fitting procedure and its assumptions. Nor does a robust residual loss guarantee protection against high-leverage predictor values.

Choosing and Reporting the Distinction

The appropriate choice depends on the concern. When the question is whether unequal error variances make conventional OLS standard errors unreliable, HC standard errors preserve the OLS fit while adjusting its estimated covariance. When the concern is that large residuals may strongly influence the fitted coefficients, a robust regression loss changes the fit. These concerns can occur together, but addressing one does not automatically address the other.

A confidence interval or test based on robust standard errors still uses the OLS coefficient as its center. The standard error affects the interval width and test statistic, not the coefficient itself. In sufficiently large samples, such inference is commonly based on an approximate normal or \(t\) reference distribution; small samples can require additional care. For robust regression, do not attach the ordinary OLS standard error automatically: inference must match the estimator and its assumptions.