Why a Preliminary Test Can Change the Answer
In a linear regression, a heteroskedasticity diagnostic can be useful for identifying a possible variance problem. A tempting next step is to use classical standard errors if the diagnostic does not reject homoskedasticity and robust standard errors if it does. That rule seems to let the data decide which method is appropriate. But the diagnostic and the coefficient inference are calculated from the same sample, so the choice itself can affect the behavior of the final test or interval.
The central problem is not that heteroskedasticity diagnostics are useless. It is that a failure to reject a diagnostic null is not proof that the null is true, and using that result to select an inferential procedure introduces an additional data-dependent decision. The resulting inference does not automatically retain the nominal error rate of either procedure.
The Assumptions Behind the Two Choices
Consider the regression model \(y_i=x_i'\beta+u_i\), for observations \(i=1,\ldots,n\), where \(x_i\) is the regressor vector and \(\beta\) is the coefficient vector. For the discussion here, assume the conditional mean is correctly specified, \(E(u_i\mid X)=0\), and observations satisfy the independence and regularity conditions needed for the stated large-sample results. The focus is the conditional variance, not changes to the regression mean.
Under homoskedasticity, \(\operatorname{Var}(u_i\mid X)=\sigma^2\), the classical covariance estimator for OLS is \(\hat{\sigma}^2(X'X)^{-1}\). Under conditional heteroskedasticity, the OLS sampling variance instead depends on the observation-specific conditional variances. The heteroskedasticity-robust sandwich estimator, such as HC0, estimates this variance without requiring a common conditional variance. Its usual justification is asymptotic; it is not an exact finite-sample guarantee.
These assumptions matter for interpretation. Classical standard errors rely on homoskedasticity for their usual covariance formula. Robust standard errors allow conditional variance to vary, but they do not repair a misspecified conditional mean or dependence between observations. Nor does a robust estimate necessarily perform better in every finite sample. A test for heteroskedasticity can provide evidence about the variance structure, but it cannot certify that one covariance formula is correct.
How Selection Enters the Final Error Rate
Let \(D=1\) mean that the preliminary diagnostic rejects its null, and \(D=0\) mean that it does not. Let \(R=1\) mean that the final coefficient test rejects its null. If the procedure chooses the standard error based on \(D\), the overall rejection probability is
This identity follows by partitioning the rejection event into the disjoint cases \(D=1\) and \(D=0\). It shows why the nominal level of the final test cannot simply be assumed after pretesting. For the overall rejection probability to equal \(\alpha\), the weighted conditional rejection probabilities in the display must combine to \(\alpha\). That equality does not follow merely because each method is described as a level-\(\alpha\) procedure in its intended setting.
The diagnostic and coefficient test generally use the same observations and residuals. Consequently, the diagnostic decision and the final test statistic need not be independent. Even if a procedure has a nominal level under a particular fixed covariance method and model, selecting between procedures based on the data can change its conditional and overall behavior. The distortion can be upward or downward; it is not universally conservative or universally anti-conservative.
Worked Example: A Non-Rejecting Diagnostic Selects a Different Conclusion
Suppose a hypothetical regression has 80 observations. A Breusch–Pagan diagnostic gives \(p=0.08\), and the analyst set the diagnostic significance level to \(\gamma=0.05\). For a coefficient measured in outcome units per one-unit increase in a regressor, the estimate is \(0.42\). Its classical standard error is \(0.18\), while its robust standard error is \(0.22\). Test \(H_0:\beta_j=0\) against a two-sided alternative using a \(t\) approximation with \(df=78\).
The pretest does not reject because \(0.08>0.05\), so the stated rule selects the classical standard error. The classical statistic is
With 78 degrees of freedom, the two-sided p-value is approximately \(0.022\). At a final-test level of \(\alpha=0.05\), this selected analysis rejects \(H_0\).
Using the robust standard error instead gives
The corresponding two-sided \(t_{78}\) p-value is approximately \(0.060\), so this analysis does not reject at \(\alpha=0.05\). The point estimate is unchanged; the difference in the decisions comes from the estimated uncertainty. The example does not establish that the robust result is always preferable. It shows that a diagnostic p-value just above its cutoff can determine which inferential conclusion is reported, even though it does not demonstrate that homoskedasticity holds.
Diagnostic Power and the Cost of a Miss
The diagnostic’s power is the probability that it rejects its null under a specified alternative. Power depends on the alternative variance pattern, sample size, regressor design, and test. A diagnostic can have low power against some forms of heteroskedasticity, even if it is useful against others. Therefore, a non-rejection can occur frequently when the variance is not constant. If the pretesting rule then selects classical standard errors, the procedure may send many such samples down the path whose covariance assumptions are violated.
The direction of the resulting error is not determined just by saying “heteroskedasticity is present.” The effect on a particular coefficient’s estimated variance depends on the variance pattern and the design matrix. The important point is that failure to detect the pattern does not eliminate this issue.
Worked Example: What Low Diagnostic Power Means for Selection
Suppose a diagnostic has significance level \(\gamma=0.05\) and power \(0.30\) against a particular heteroskedastic alternative in a specified regression design. Imagine 200 hypothetical repeated samples generated under that alternative. The pretesting rule selects robust inference when the diagnostic rejects and classical inference otherwise.
The probability of failing to reject under this alternative is \(1-0.30=0.70\). Thus, the expected number of samples that fail to detect the variance pattern is
The expected number that detect it is \(200(0.30)=60\). These are expected counts, not guaranteed outcomes in a particular set of 200 samples. Under the stated rule, approximately 140 samples on average would select classical standard errors despite being generated under the alternative. This calculation does not say that every selected classical interval must fail; it shows why a low-powered diagnostic cannot be treated as a reliable gatekeeper for inference.
Repeated Diagnostic Decisions
Another concern arises when an analyst tries several diagnostics, variance specifications, or significance cutoffs and then reports whichever decision supports the preferred standard-error choice. Each additional look creates another opportunity for a false rejection when the null is true. The combined probability depends on the dependence among the tests and on the decision rule; a single-test significance level does not describe the entire sequence.
Worked Example: Three Independent Diagnostic Tests
For illustration, suppose three tests are each conducted at level \(0.05\), all three null hypotheses are true, and the test decisions are independent. The probability that none rejects is the product of their non-rejection probabilities:
Therefore, the probability of at least one rejection is
So the rule flags a variance concern in about \(0.1426\), or \(14.26\%\), of such samples under the stated independence assumption, rather than in \(5\%\). This calculation is not automatically applicable to real regression diagnostics: tests computed from the same data are often dependent. It demonstrates why repeatedly testing and acting on any rejection cannot be described as a single \(5\%\) decision without accounting for the sequence.
A More Reliable Inferential Strategy
A more defensible approach is to decide on the inferential strategy using the study design, substantive knowledge, and analysis goals rather than letting a diagnostic p-value mechanically choose the standard error. If conditional heteroskedasticity is plausible and the independent-observation assumptions are appropriate, robust standard errors can be specified in advance as the primary method. This avoids making the covariance choice hinge on whether a preliminary test happened to cross a cutoff.
A variance diagnostic can still be reported as a diagnostic, with its tested null and limitations stated clearly. Researchers may also report classical and robust results as a sensitivity comparison, provided they identify the prespecified primary analysis and do not present the more favorable result as though it had been chosen independently of the data. In either case, report the coefficient estimate, standard error, test or interval, and the covariance method used.
Robust inference is not a universal remedy: its usual justification is large-sample, and its finite-sample behavior can be affected by leverage and sample size. Earlier material on HC corrections addresses some of those issues. The relevant lesson here is narrower: pretesting is not a substitute for choosing assumptions and an inferential procedure deliberately.
Common Mistakes
- “The diagnostic did not reject, so the errors are homoskedastic.” A non-rejection is not proof. The test may have limited power against the variance pattern in the data.
- “The selected method has its advertised level, so the two-stage procedure does too.” Selection can make the final statistic’s behavior differ from that of either fixed method. The overall error rate must account for the selection event.
- “Robust standard errors always solve the problem.” Robust covariance estimation addresses conditional heteroskedasticity under its conditions; it does not make inference exact in every sample or remedy dependence and conditional-mean misspecification.
- “Three tests at 5% still make one 5% decision.” Not necessarily. Under independence, three level-\(0.05\) tests have probability \(0.1426\) of at least one rejection when all nulls hold; dependence changes the calculation.
Check Your Understanding
Answer each question using the distinctions and calculations in this tutorial.
- What does a failure to reject a heteroskedasticity diagnostic establish, and what does it not establish?
- In the selection identity for \(P(R=1)\), why can the overall rejection probability differ from the nominal level of a fixed-method test?
- A diagnostic has power \(0.40\) against a specified alternative. What fraction of samples generated under that alternative fail to reject, and what could the pretesting rule select for those samples?
- Under independent tests, calculate the probability of at least one false rejection from four tests, each conducted at level \(0.05\).
- Name one reason robust standard errors are not a universal solution to every inference problem in regression.