Weak-Strength Diagnostics for Binary Instruments

Why Binary Instruments Need Their Own Strength Check

When an instrument \(Z\) takes only the values 0 and 1, its unadjusted association with a binary treatment \(D\) has a direct interpretation: the difference in treatment proportions between the two instrument groups. With covariates, the relevant question becomes whether \(Z\) still predicts \(D\) after accounting for those covariates. These are two views of first-stage relevance, not tests of whether the instrument is valid.

This distinction matters because a large difference in treatment rates can be driven partly by different covariate compositions across instrument groups. Conversely, adjustment can reveal a substantial within-covariate association that a crude comparison obscures. A careful diagnostic reports the model, the test, and a measure of incremental explanatory power rather than treating one statistic as a universal verdict.

The linear model is commonly used for first-stage diagnostics even when \(D\) is binary. It is a linear probability model: fitted values are not guaranteed to lie between 0 and 1, but the coefficient on \(Z\) has a useful scale, namely a change in treatment probability. The diagnostic concerns prediction in this specified first-stage regression. It does not by itself establish the exclusion restriction, independence, or any other requirement for a valid instrument.

Unadjusted Group Differences

With no covariates, the ordinary least-squares coefficient on a 0/1 instrument in a regression of \(D\) on an intercept and \(Z\) equals the difference between the two sample treatment proportions. Define \(\bar{D}_1\) as the mean of \(D\) among observations with \(Z=1\), and \(\bar{D}_0\) as the mean among those with \(Z=0\). Then the estimated first-stage difference is \(\hat{\pi}=\bar{D}_1-\bar{D}_0\). Its sign shows the direction of the observed relationship, and its magnitude is in treatment-probability units.

A group difference can be informative, but its standard error depends on the inferential design. The simple proportion formula above assumes independent observations and uses a large-sample approximation. If observations are clustered, sampled with a complex design, or require other dependence adjustments, use an appropriate variance estimator rather than this formula.

Adjustment, Nested Tests, and Partial R-Squared

To assess relevance conditional on covariates, compare two regressions for the same treatment outcome and sample. The restricted model contains an intercept and \(X\); the unrestricted model adds \(Z\). If the additive specification is appropriate, the unrestricted model estimates the conditional first-stage coefficient \(\hat{\pi}\). The test \(H_0:\pi=0\) asks whether adding the instrument improves the first-stage model after the included covariates are accounted for.

Let \(SSE_R\) and \(SSE_U\) be the residual sums of squares for the restricted and unrestricted models. With one excluded instrument term, the conventional partial F-statistic is

where \(df_U\) is the residual degrees of freedom in the unrestricted model. Under the classical linear-model assumptions of independent, homoskedastic errors and the specified model, this statistic has an \(F_{1,df_U}\) distribution under \(H_0\). For one added regressor, it equals the square of the conventional \(t\)-statistic for \(\hat{\pi}\).

Partial R-squared measures the fraction of the restricted model’s residual variation in \(D\) that is removed by adding \(Z\). It is conditional on the included covariates; it is not the overall R-squared of the first-stage regression.

Why the Partial F and Partial R-Squared Are Linked

The connection follows directly from the two definitions. Write \(R^2_{\mathrm{partial}}=(SSE_R-SSE_U)/SSE_R\), so \(SSE_R-SSE_U=R^2_{\mathrm{partial}}SSE_R\), while \(SSE_U=(1-R^2_{\mathrm{partial}})SSE_R\). Substituting both expressions into the conventional test gives

Thus the test statistic reflects both the incremental explanatory power and the residual degrees of freedom. The same partial R-squared can yield a larger F-statistic in a larger sample. A small p-value is evidence against the zero-coefficient null under the test’s assumptions; it is not a measure of instrument validity or, by itself, a complete assessment of weak-instrument risk.

Covariate Composition Can Change the Picture

The crude group difference and the covariate-adjusted coefficient need not be equal. If the distribution of \(X\) differs between instrument groups, the crude difference combines within-covariate differences with differences in group composition. The next example makes this visible and also calculates the partial test from the corresponding regression sums of squares.

Model Choices and Failure Modes

The model used for adjustment must match the diagnostic question. In the additive model, the instrument has one common coefficient across covariate values. If the analysis includes interactions between \(Z\) and \(X\), relevance is no longer summarized by a single coefficient: a joint test should test that all excluded-instrument terms, including the interactions, are zero. For \(q\) such terms, the conventional partial F-statistic divides the reduction in \(SSE\) by \(q\), then divides by the unrestricted residual mean square. Partial R-squared uses the same total reduction, \((SSE_R-SSE_U)/SSE_R\). The simple identity shown above applies with one added term; with multiple terms, \(F=[R^2_{\mathrm{partial}}/q]/[(1-R^2_{\mathrm{partial}})/df_U]\).

The conventional nested-model F-test relies on its variance assumptions. If errors are heteroskedastic or observations are clustered, use a suitable robust or cluster-robust Wald test for the excluded instrument terms. Do not assume that a conventional partial F-statistic or its simple relationship to partial R-squared remains valid with a robust variance estimator. Report which test was used and the assumptions supporting it.

Useful diagnostic checks include verifying the instrument’s coding and group counts, confirming that the restricted and unrestricted models use the same observations and covariates, and checking whether the instrument varies within relevant covariate strata. If \(Z\) is nearly predictable from \(X\), there may be little independent variation available to estimate its conditional association with treatment. Sparse cells can also make estimates and standard errors unstable. A single average coefficient can conceal weak or differently directed relationships across subgroups when effects vary.

How to Interpret the Evidence

A group difference is the simplest descriptive measure. The covariate-adjusted coefficient answers a conditional first-stage question under the chosen model, while a partial F-test assesses whether the excluded instrument terms improve fit relative to a specified restricted model. Partial R-squared gives a scale-free measure of that incremental fit. These quantities are complementary: a p-value depends on precision and sample size, whereas partial R-squared describes the fraction of remaining variation explained.

There is no universal cutoff that turns a first-stage statistic into a guarantee of reliable conventional IV inference. In particular, a familiar rule of thumb for an F-statistic should not be treated as a pass/fail boundary that applies regardless of the number of instruments, variance estimator, design, or inferential goal. Weak-instrument diagnostics assess the strength of the first-stage relationship; they do not repair invalid instruments. Weak-IV-robust procedures address inference differently, but a diagnostic alone does not replace them when strength is a concern.