When Robust and Conventional Conclusions Differ

Introduction

A regression coefficient is an estimate of a relationship in the fitted model; its standard error describes uncertainty about that estimate. When the error variance is unequal across observations, a conventional standard error and a heteroskedasticity-robust standard error can give different accounts of that uncertainty. As a result, their confidence intervals may have different widths, and their tests may lead to different decisions.

The comparison is useful precisely because the fitted coefficients do not change. It isolates what follows from changing the estimated uncertainty while holding the fitted model and its coefficient estimates fixed. A difference in statistical significance is not a change in the estimated relationship; it is a change in how uncertain that estimate appears under the two variance calculations.

What Changes—and What Does Not

For a regression with \(n\) observations and \(p\) fitted coefficients, write the model as \(Y=X\beta+\varepsilon\). Here, \(Y\) is the response vector, \(X\) is the design matrix, \(\beta\) is the vector of coefficients, and \(\varepsilon\) is the error vector. Assume \(X\) has full column rank. The least-squares estimate is

Once \(X\) and \(Y\) are fixed, this estimate is fixed. The conventional and robust procedures do not refit the model or alter the observed responses and predictors; they differ in their estimate of the covariance matrix of \(\hat{\beta}\). Under homoskedastic errors, the conventional covariance estimate has the form \(\hat{\sigma}^2(X^\mathsf{T}X)^{-1}\). A heteroskedasticity-robust estimate uses a sandwich form, with residual-based contributions that allow different observations to have different error variances. Variants such as HC3 modify those contributions to account for leverage.

The standard error for a particular coefficient is the square root of the corresponding diagonal entry of the covariance estimate. A different covariance estimate can therefore change the standard error without changing the coefficient. For a test of \(H_0:\beta_j=0\), the estimate divided by its standard error gives the test statistic. The coefficient is the same in both calculations; its standard error is not.

From Standard Errors to Intervals and Tests

A confidence interval combines a coefficient estimate with a critical value and a standard error. For a coefficient \(\beta_j\), a two-sided interval at confidence level \(1-\alpha\) has the form

Here \(t^*\) is the chosen reference-distribution critical value. For a two-sided test of \(H_0:\beta_j=0\), the test statistic is

A larger standard error makes the interval wider and, for a fixed nonzero estimate, makes the absolute test statistic smaller. A smaller standard error has the reverse effects. Whether either change alters the final decision depends on how close the original result was to the chosen cutoff.

The conventional formulas rely on homoskedastic errors for their usual variance calculation. Robust standard errors are intended to protect the covariance estimate against unequal error variances, provided the regression mean is appropriately specified and the observations meet the relevant independence conditions. They do not make every assumption unnecessary, and they do not change the regression model itself.

A Short Justification for the Unchanged Coefficients

The coefficient formula depends on the observed design matrix and response vector, not on which covariance estimate is later used. Under the homoskedastic model, \(\operatorname{Var}(\varepsilon\mid X)=\sigma^2I\), the least-squares covariance is

If instead \(\operatorname{Var}(\varepsilon\mid X)=\Omega\), where the diagonal entries of \(\Omega\) need not be equal, the same calculation gives

The expression for \(\hat{\beta}\) is unchanged; the covariance expression is what differs. A robust covariance estimator approximates this latter form without requiring the unequal variances to follow one specified pattern. This explains why robust inference can revise uncertainty while leaving the fitted coefficients exactly as they were.

Worked Comparisons

The following examples are hypothetical. Each uses a fitted simple linear regression based on 50 independent observations, so the conventional residual degrees of freedom are \(50-2=48\). For direct comparison, the examples use a \(t\) reference distribution with 48 degrees of freedom for both calculations. Robust procedures are often justified by large-sample approximations; the reference choice and its limitations should be made explicit in an actual analysis.

How to Compare Conclusions Responsibly

When the conventional and robust results differ, report the comparison rather than treating one \(p\)-value as a replacement for the other without explanation. State the coefficient estimate, the standard error method, the interval or test result, and the significance level. An interval describes values reasonably compatible with the estimate under the stated procedure; it does not assign a probability to the fixed coefficient being inside that particular interval.

A robust standard error is not automatically larger. As the examples illustrate, robust uncertainty can be greater or smaller than conventional uncertainty. Nor does a change in statistical significance imply that the estimated relationship has changed. It indicates that the uncertainty calculation has changed enough to affect whether a particular threshold is crossed. The practical size and context of the coefficient remain important whether or not a test rejects \(H_0\).

Robust standard errors address a variance-estimation problem. They do not correct a poorly specified conditional mean, make dependent observations independent, or establish a causal relationship. In particular, an inference method cannot turn an association into evidence of causation without an appropriate study design and assumptions.

Common Mistakes

• Claiming the coefficient changed. The OLS coefficient estimate is calculated from \(X\) and \(Y\). Changing the covariance estimator changes its estimated uncertainty, not the fitted coefficient.
• Assuming robust standard errors must be larger. They can be larger or smaller. Compare the actual values rather than relying on a presumed direction.
• Comparing decisions without stating the level. A reject-or-not-reject conclusion depends on the chosen \(\alpha\). Always state the level and use a matching two-sided interval if using intervals to check the test.
• Treating non-rejection as proof of no relationship. Not rejecting \(H_0\) means the test did not provide sufficient evidence against the null at the specified level; it does not establish that the coefficient is exactly zero.
• Assuming robust standard errors fix every problem. They address unequal error variance in the covariance calculation. They do not by themselves repair model misspecification or dependence among observations.