Robust First-Stage Diagnostics and Effective F Statistics

Why Robust First-Stage Diagnostics Are Needed

In instrumental-variables analysis, a first stage asks whether the excluded instruments predict the endogenous treatment after accounting for included covariates. A strong relationship is important: when instruments have little predictive power, conventional 2SLS estimates and standard inference can be unreliable. But a diagnostic that is appropriate under constant error variance may not be appropriate when the variance differs across observations.

A heteroskedasticity-robust first-stage test addresses that variance problem when testing whether the excluded instruments have any conditional predictive contribution. It does not, by itself, answer the more demanding question of whether they are strong enough for conventional IV inference. The Montiel Olea–Pflueger effective F statistic is designed to assess strength in settings with heteroskedastic errors and one endogenous regressor. The distinction between these two questions is central.

First-Stage Model and Robust Wald Test

Suppose there is one endogenous regressor \(X_i\), a vector of \(q\) excluded instruments \(Z_i\), and a vector of included exogenous controls \(W_i\). The first-stage linear model is

X_i = W_i'\gamma + Z_i'\pi + v_i.

Here \(\pi\) is the vector of coefficients on the excluded instruments. The first-stage relevance null is \(H_0:\pi=0\), meaning that the excluded instruments have no linear predictive contribution conditional on the included controls. The alternative is \(H_a:\pi\ne 0\). This is a joint test: with \(q\) instruments, it tests all \(q\) coefficients together.

For a heteroskedasticity-robust test, estimate the first stage and use a heteroskedasticity-robust covariance matrix for \(\hat{\pi}\). In words, this covariance estimator allows the first-stage error variance to differ across observations, while relying on observations being independent for the usual heteroskedasticity-robust large-sample justification. Included controls must be handled in both the regression and the covariance calculation.

Definition: The robust Wald statistic for the joint first-stage null is \(W=\hat{\pi}'\widehat{\mathrm{Var}}_{\mathrm{rob}}(\hat{\pi})^{-1}\hat{\pi}\), where \(\widehat{\mathrm{Var}}_{\mathrm{rob}}(\hat{\pi})\) is the heteroskedasticity-robust covariance matrix of the excluded-instrument coefficient estimates. Under \(H_0\), appropriate regularity conditions, and a sufficiently large sample, \(W\) is approximately distributed as a chi-square random variable with \(q\) degrees of freedom.

Some software reports \(W/q\) as a robust Wald F statistic. In large samples this is a rescaling of the same Wald test, commonly compared with an F reference distribution using a specified denominator degrees of freedom. The exact finite-sample reference distribution and degrees of freedom depend on the implementation. A conventional homoskedastic first-stage F statistic is not made heteroskedasticity-robust merely by labeling it an F test.

Worked Example: A Robust Joint Relevance Test

Suppose a hypothetical first-stage regression has two excluded instruments. After controlling for \(W_i\), the estimated instrument coefficients are \(\hat{\pi}_1=0.18\) and \(\hat{\pi}_2=0.12\). Their reported robust covariance matrix is diagonal, with estimated variances \(0.0036\) and \(0.0016\). Test the joint null that both coefficients are zero.

The Wald statistic is

W = \begin{pmatrix}0.18 & 0.12\end{pmatrix} \begin{pmatrix}0.0036 & 0\\0 & 0.0016\end{pmatrix}^{-1} \begin{pmatrix}0.18\\0.12\end{pmatrix} = \frac{0.18^2}{0.0036}+\frac{0.12^2}{0.0016} =9+9=18.

There are \(q=2\) restrictions, so the large-sample reference is \(\chi^2_2\). Its upper-tail probability at 18 is approximately \(0.0001234\). At the 5% significance level, reject the joint null of zero excluded-instrument coefficients. This is evidence that the instruments predict \(X_i\) conditional on the controls. It is not a test of whether the instruments are valid, nor does this p-value establish that their predictive strength is adequate for conventional IV inference.

Why Relevance and Strength Are Different

The null \(H_0:\pi=0\) is a point null: it asks whether the population first-stage coefficients are exactly zero. Weak-instrument concerns are different. They arise when the first-stage relationship is nonzero but too small, relative to its uncertainty and the sample size, to support reliable conventional IV inference. With a sufficiently large sample, even a modest first-stage relationship can produce a small p-value for the zero-coefficient null.

Thus, the robust Wald test is useful for testing conditional relevance while allowing heteroskedasticity. It is not a general-purpose measure of how much weak-instrument distortion to expect. Dividing its statistic by the number of instruments does not turn it into a calibrated weak-instrument diagnostic. The scale and reference distribution needed for a strength assessment depend on more than the rejection of a zero-coefficient null.

Worked Example: A Large Wald Statistic Does Not Settle Strength

Consider a hypothetical study with \(n=20{,}000\) observations and three excluded instruments. Suppose a robust first-stage test reports \(W=24\). The joint test has \(q=3\) restrictions, so its large-sample chi-square p-value is the upper-tail probability \(P(\chi^2_3\ge 24)\), approximately \(2.5\times 10^{-5}\). The equivalent rescaled statistic is \(W/q=24/3=8\).

The small p-value is strong evidence against the exact null \(\pi=0\). But neither \(W=24\) nor \(W/q=8\), considered without an appropriate weak-instrument calibration, establishes that conventional IV inference is reliable. The large sample can make a joint zero-coefficient test reject even when the predictive relationship is not strong enough for that purpose. A strength diagnostic calibrated for the model and error structure is needed for that separate assessment.

The Montiel Olea–Pflueger Effective F Statistic

The Montiel Olea–Pflueger (MOP) effective F statistic, denoted \(F_{\mathrm{eff}}\), is a heteroskedasticity-robust diagnostic of instrument strength for a linear IV model with one endogenous regressor and one or more excluded instruments. It uses information about the first-stage signal and its sampling uncertainty to form a statistic for assessing strength. In contrast with a test of \(H_0:\pi=0\), it is intended to be interpreted against critical values calibrated to a specified weak-instrument distortion criterion.

Definition: \(F_{\mathrm{eff}}\) is the Montiel Olea–Pflueger effective F statistic for assessing the strength of excluded instruments for one endogenous regressor when heteroskedasticity is allowed. Its value is a diagnostic statistic, not a p-value. To use it as a test of adequate strength, compare it with a critical value appropriate to the chosen distortion tolerance and applicable assumptions.

The effective F is not generally the robust Wald statistic divided by \(q\), and it need not equal the conventional first-stage F statistic. Its heteroskedasticity-robust construction accounts for the fact that the uncertainty in estimated first-stage coefficients need not have the simple constant-variance form used by the conventional homoskedastic statistic. In practice, use a documented implementation of the MOP statistic, with the intended heteroskedasticity-robust settings, rather than trying to substitute a robust first-stage p-value for \(F_{\mathrm{eff}}\).

The MOP framework calibrates strength thresholds to a stated criterion, such as bounding the relative bias of 2SLS compared with OLS or limiting size distortion in conventional Wald inference. The statistic alone does not specify which criterion is being used, and it has no universal cutoff that applies in every analysis. The choice of calibration and its assumptions matter; a critical value is not a property of the observed statistic alone.

Worked Example: Interpreting an Effective F Output

Suppose a hypothetical independent-observation study has one endogenous treatment, four excluded instruments, and included exogenous controls. The first-stage diagnostic software reports \(F_{\mathrm{eff}}=8.7\). A separate heteroskedasticity-robust joint test reports \(W=16.2\), so its rescaled value is \(16.2/4=4.05\).

These reported numbers answer different questions. The robust joint test assesses \(H_0:\pi=0\); its statistic is \(W=16.2\) on four restrictions. The MOP output, \(F_{\mathrm{eff}}=8.7\), is the strength diagnostic to compare with an MOP critical value calibrated to the analysis’s chosen distortion criterion. The value 8.7 is not a p-value, and it should not be compared with an arbitrary rule of thumb. Without specifying the calibration and its corresponding critical value, the conclusion is that the statistic has been calculated, not that instruments have passed or failed a strength test.

The difference between 8.7 and 4.05 is not an arithmetic inconsistency: the rescaled Wald statistic and the effective F are distinct statistics. Both use first-stage information, but they are constructed and interpreted for different purposes.

Justification for the Robust Wald Reference

The large-sample reference for the robust Wald test follows from the sampling behavior of the first-stage coefficient estimates. Under the joint null and regularity conditions, the robust covariance estimator consistently estimates the covariance of \(\hat{\pi}\). Consequently, the vector standardized by its estimated covariance approaches a \(q\)-dimensional standard normal vector. Its squared length is the Wald statistic.

\widehat{\mathrm{Var}}_{\mathrm{rob}}(\hat{\pi})^{-1/2}\hat{\pi} \ \overset{d}{\longrightarrow}\ N(0,I_q) \quad\Longrightarrow\quad W = \left\|\widehat{\mathrm{Var}}_{\mathrm{rob}}(\hat{\pi})^{-1/2}\hat{\pi}\right\|^2 \ \overset{d}{\longrightarrow}\ \chi^2_q.

The conclusion uses large-sample approximation, not an exact finite-sample identity. Robust covariance estimation protects this test against heteroskedasticity under its assumptions; it does not make the instruments exogenous, eliminate dependence across observations, or convert a zero-coefficient test into a weak-instrument test.

Conditions, Scope, and Common Pitfalls

For these diagnostics to be useful, the first stage must be specified consistently with the IV model: include the same exogenous controls and excluded instruments that enter the analysis, and use the same sample. The ordinary heteroskedasticity-robust covariance estimator assumes independent observations. If observations are clustered or otherwise dependent, the covariance estimator and diagnostic must account for that dependence; simply selecting a heteroskedasticity-robust option is not a substitute for a dependence-robust procedure.

The MOP effective F discussed here applies to one endogenous regressor. Do not transfer its interpretation or critical values automatically to a model with multiple endogenous regressors. Also keep strength separate from instrument validity: first-stage diagnostics describe predictive strength, not whether instruments satisfy the causal assumptions needed for IV estimation.

A common mistake is to reason, “The robust first-stage p-value is below 0.05, so the instruments are strong.” The p-value addresses the exact null of no conditional first-stage relationship. It does not quantify whether conventional IV inference has acceptable bias or size distortion. The correction is to report the robust relevance test for its own purpose and use an appropriately calibrated strength diagnostic, such as \(F_{\mathrm{eff}}\), for the weak-instrument question. Report the statistic, the calibration criterion and critical value used, and the assumptions supporting the calculation.

Key takeaway: A heteroskedasticity-robust Wald test asks whether excluded instruments jointly predict the endogenous regressor. The MOP effective F assesses strength for one endogenous regressor using a distortion-based calibration. Neither statistic tests instrument validity, and a small p-value for relevance is not proof of adequate strength.

Check Your Understanding

Answer each question by distinguishing the hypothesis or diagnostic involved.

With three excluded instruments, what joint null does the robust first-stage Wald test assess?
Why can a very small robust first-stage p-value fail to show that instruments are strong enough for conventional IV inference?
What does the MOP effective F statistic assess, and why is it not itself a p-value?
Why should a robust first-stage Wald statistic divided by the number of instruments not automatically be reported as \(F_{\mathrm{eff}}\)?
Name two considerations that must be checked before interpreting an effective F statistic or its critical value.