Sample Size and Power for Instrument Diagnostics

Why Power Belongs in Instrument Diagnostics

A first-stage diagnostic asks whether the excluded instrument or instruments predict the endogenous treatment after accounting for included covariates. Its power is the probability that the diagnostic rejects the null of no first-stage association under a specified alternative. Power therefore depends on more than the number of observations: the size of the first-stage association, the variation in the instruments, the number of instruments, and the dependence structure all matter.

A diagnostic can have low power even when an instrument is relevant, and a rejection does not establish the exclusion restriction or instrument exogeneity. Those are substantive identifying assumptions, not conclusions supplied by a first-stage test. The planning question is narrower: under plausible design and effect scenarios, how often would the chosen diagnostic detect a specified first-stage association?

Model and the Source of First-Stage Power

Let \(D_i\) be the endogenous treatment, \(Z_i\) a vector of \(q\) excluded instruments, and \(X_i\) a vector of included covariates. A linear first-stage model is

Here \(\pi\) is the vector of first-stage coefficients. The usual joint diagnostic tests \(H_0:\pi=0\), conditional on \(X_i\). The standard homoskedastic F reference distribution requires, among other conditions, a correctly specified linear conditional mean, independent observations, and errors that are homoskedastic and normally distributed for exact finite-sample calibration. With large samples, related approximations can apply under weaker conditions. If errors are heteroskedastic or clustered, the test statistic and its reference distribution must use an appropriate robust covariance estimator; the ordinary F calculation need not be valid.

For a fixed design, let \(Z\) and \(X\) be the matrices of instruments and covariates, and let \(M_X=I-X(X'X)^{-1}X'\) project away the included covariates. Under homoskedastic first-stage errors with variance \(\sigma_v^2\), the noncentrality parameter for the joint excluded-instrument test is

Under the alternative, the F statistic has a noncentral F distribution with numerator degrees of freedom \(q\), the appropriate residual degrees of freedom in the denominator, and noncentrality parameter \(\lambda_{\mathrm{FS}}\). Thus its power at level \(\alpha\) is the probability that this noncentral F statistic exceeds the central-F critical value. The formula makes the design dependence explicit: more observations help only insofar as they add useful residualized instrument variation, and a larger \(\sigma_v^2\) makes the same first-stage signal harder to distinguish from noise.

For one instrument, no covariates, and independent observations with instrument variance \(\operatorname{Var}(Z_i)\), the noncentrality is approximately

The second expression is the approximate noncentrality of the first-stage coefficient's t statistic. It shows why sample size alone is not enough to specify power: the instrument's variance and the first-stage error scale are also required. If covariates are included, use the instrument variation remaining after projecting on those covariates, not its unadjusted variance.

Power for Weak-IV-Robust Inference

A first-stage test and a weak-IV-robust test answer different questions. The former tests whether the instruments predict treatment. An Anderson–Rubin (AR) test evaluates a hypothesized treatment effect while retaining validity under weak first-stage relevance, when its specified error and covariance assumptions hold. Weak-IV robustness protects size; it does not promise high power. With little instrument variation or a small reduced-form effect, AR inference can be valid but imprecise.

To see the signal involved, consider the structural model \(Y_i=\beta D_i+X_i'\gamma+\varepsilon_i\), alongside the first stage above. For testing \(H_0:\beta=\beta_0\), form the outcome contrast \(Y_i-\beta_0D_i\). Substitution gives

Under \(H_0\), the excluded-instrument coefficient in this regression is zero. Under an alternative, its signal is \(\pi(\beta-\beta_0)\). In a simplified one-instrument design with independent observations, residualized instrument variance \(\operatorname{Var}(Z_i)\), and contrast-error standard deviation \(\sigma_{\mathrm{AR}}\), the approximate standardized signal is

Here \(\sigma_{\mathrm{AR}}\) is the standard deviation of \(\varepsilon_i+(\beta-\beta_0)v_i\), under the alternative being considered. It cannot generally be replaced by the standard deviation of \(\varepsilon_i\) without an additional assumption. For multiple instruments, power depends on the joint instrument design and the covariance used for the test. Robust or clustered AR procedures require their corresponding variance calculations rather than this simple independent-observation approximation.

Instrument Count and Dependence

With \(q\) instruments, the joint test has \(q\) numerator degrees of freedom. If the additional instruments carry independent, relevant variation after accounting for \(X\), they can increase \(\lambda_{\mathrm{FS}}\). But adding instruments also changes the reference distribution and its critical value. An added instrument with no conditional relevance may contribute no signal while increasing the numerator degrees of freedom; consequently, adding instruments does not guarantee higher diagnostic power. Correlation among instruments can also make their combined residualized variation less informative than their count suggests.

Dependence changes the amount of independent information. For clustered observations, within-cluster correlation generally increases uncertainty relative to an independent-observation calculation. A rough design-effect illustration for equal cluster size \(m\) and exchangeable error correlation \(\rho\) is \( DE=1+(m-1)\rho \). In settings where this approximation is appropriate, a rough effective sample size is \(n/DE\). It is only a planning heuristic: cluster-level instrument assignment, unequal cluster sizes, covariate adjustment, and the actual covariance pattern can make this substitution inaccurate. Inference must reflect the real clustering, and the number of independent clusters can be more consequential than the total number of individuals.

Planning, Diagnostics, and Limitations

Power planning is most useful as a sensitivity analysis across plausible first-stage coefficients, residual variation, instrument variation, sample sizes, and dependence structures. For a joint F diagnostic, calculate power from the intended noncentral F model when its assumptions are credible. For robust or clustered tests, use a method calibrated to that covariance structure. For weak-IV-robust inference, plan power for scientifically meaningful alternatives in \(\beta\), not merely power to reject a zero first stage.

A useful planning sequence is:

After data collection, the observed first-stage statistic is a diagnostic, not a measure of the probability that the instrument is valid. A large statistic does not verify exclusion or exogeneity. Nor should an observed statistic be used to calculate “post-hoc power” as though it supplied new evidence about the design: it largely restates the observed result. Report the diagnostic, its assumptions and covariance method, and relevant uncertainty. Weak-IV-robust confidence sets or tests remain important when conventional inference may be unreliable.

The conventional first-stage F statistic is a familiar summary, but a single threshold is not a universal guarantee of reliable IV inference. It does not encode every design feature, such as multiple correlated instruments or clustering. AR procedures provide inference valid under weak relevance under their own assumptions, but can have low power and yield wide, unbounded, or disconnected confidence sets. The appropriate comparison is therefore not “diagnostic versus robust inference”: diagnostics describe evidence about first-stage association, while weak-IV-robust methods address uncertainty about the treatment effect.

Common Mistakes

• Using sample size without instrument variation. A signal calculation needs the residualized instrument variance as well as \(n\), the coefficient, and the error scale.
• Assuming more instruments always increase power. Additional instruments can add signal, but also change degrees of freedom and may contribute little conditional variation.
• Treating a first-stage rejection as proof of validity. Relevance is not the same as exclusion or exogeneity.
• Ignoring dependence. Applying an independent-observation formula to clustered data can overstate effective information and miscalibrate inference.
• Confusing weak-IV robustness with high power. A test can control its rejection rate under weak instruments while still having low power against plausible alternatives.