Practical Checks On Instrument Validity

Introduction

An instrumental variable can produce a precise estimate and still be an unconvincing instrument. Its credibility depends on how its variation was generated and whether that variation could affect the outcome through channels other than the endogenous treatment. Researchers therefore combine design documentation with empirical checks of observed data. These checks can expose problems or add support to an argument, but they cannot directly reveal every pathway linking an instrument to an outcome.

The distinction matters: a small p-value in a balance check may raise a concern, while a large p-value does not certify validity. The central restriction concerns unobserved determinants of the outcome and the instrument’s possible direct effects. Neither is fully settled by inspecting a collection of observed variables.

The IV Conditions Being Assessed

Let \(Y\) be an outcome, \(D\) an endogenous treatment or exposure, \(Z\) an instrument, and \(X\) a vector of exogenous controls that includes an intercept. Consider the linear structural equation

Y=\beta D+X'\gamma+u.

Here \(\beta\) is the structural effect of interest under the model, \(\gamma\) is the coefficient vector for the controls, and \(u\) collects unobserved determinants of \(Y\). The instrument must be relevant for \(D\) after accounting for \(X\), and it must satisfy an appropriate exclusion and exogeneity restriction. Relevance means that the instrument has variation associated with treatment conditional on the controls; without it, the instrument does not provide the identifying variation required for IV.

Definition: In this setting, a valid instrument is relevant for \(D\) conditional on \(X\) and satisfies the required exclusion and exogeneity restriction. A commonly used conditional mean restriction is \(E[u\mid Z,X]=E[u\mid X]\). A weaker, distinct IV moment restriction is \(E[(Z-E[Z\mid X])u]=0\). The conditional mean restriction implies this moment, but the moment alone does not imply conditional mean independence.

Exclusion means that the instrument does not affect \(Y\) through a direct route outside the modeled treatment \(D\). Exogeneity means that the instrument’s identifying variation is not systematically related to the unobserved determinants represented by \(u\), in the sense imposed by the chosen model. These ideas are related but not interchangeable: institutional knowledge helps assess the possible pathways, while the moment restriction states the statistical condition used for identification.

Why an IV Moment Is Not a Complete Empirical Test

The distinction between conditional mean independence and a moment condition can be seen directly. Write \(m(X)=E[Z\mid X]\). If \(E[u\mid Z,X]=E[u\mid X]\), then iterated expectations give

\begin{aligned} E[(Z-m(X))u] &=E\{E[(Z-m(X))u\mid X]\}\\ &=E\{E[u\mid X]E[Z-m(X)\mid X]\}\\ &=0. \end{aligned}

The last equality holds because \(E[Z-m(X)\mid X]=0\). This argument establishes an implication, not an equivalence. For a counterexample, let \(X\) be constant, let \(Z\) take values \(-1,0,1\) with equal probability, and let \(u\) take values \(1,-2,1\) at those respective values of \(Z\). Then

E[(Z-E[Z])u]=\frac{(-1)(1)+(0)(-2)+(1)(1)}{3}=0, \qquad E[u\mid Z]\in\{1,-2,1\}.

The unconditional moment is zero, yet the conditional mean of \(u\) varies with \(Z\). Thus, evidence consistent with a particular moment does not establish the stronger conditional statement. In practice, \(u\) is not observed as a separate data column: it is the structural error in the equation. This is one reason that empirical checks can diagnose implications or observable consequences of a design, but cannot certify the full validity condition.

Start With the Design Record

Design documentation addresses a question that regression tables cannot answer on their own: where did the instrument’s variation come from? Relevant evidence may include assignment protocols, dated policy documents, administrative rules, implementation records, and information about who could influence assignment. A persuasive account should specify the population, the assignment mechanism, the timing, and the channels through which the instrument might affect the outcome.

For example, a claim that program offers were assigned by lottery is more credible when a dated protocol explains the lottery, archived records show that offers followed it, and staff could not override assignments. The account is less persuasive if eligibility was changed after outcomes were known, staff selectively made offers, or the offer also provided a separate service that could directly affect the outcome. Documentation does not prove that every relevant feature was recorded, but it can make the proposed source of variation concrete and open to scrutiny.

A practical review asks whether the instrument preceded treatment and outcome measurement, whether the assignment rule was applied as described, whether there were exceptions, and whether the instrument could alter the outcome through another route. It also asks whether the analysis conditions on variables determined after assignment. Such controls can change the comparison being made and may themselves be affected by the instrument; their inclusion should have a design-based rationale rather than being chosen only because they improve a result.

Balance Checks on Predetermined Characteristics

A balance check asks whether the instrument is associated with measured characteristics that were determined before the instrument could affect them. For a binary instrument, a simple check compares the mean of a baseline covariate between the \(Z=1\) and \(Z=0\) groups. With controls, the analogous check examines whether the instrument predicts the baseline covariate after accounting for those controls. These checks are diagnostics of the assignment story: imbalance can signal sorting, implementation problems, or chance variation.

Balance is not a mechanical pass-or-fail test of IV validity. A design can have an imbalance by chance, especially when many characteristics are checked. Conversely, measured characteristics may be balanced even when the instrument is related to unobserved determinants. A standardized difference can help describe the size of an imbalance alongside its uncertainty:

\text{standardized difference} = \frac{\bar{W}_{Z=1}-\bar{W}_{Z=0}}{s_{\text{pooled}}}, \qquad s_{\text{pooled}} = \sqrt{\frac{s_1^2+s_0^2}{2}}.

Worked Example: A Baseline Balance Difference

In a hypothetical evaluation, \(Z=1\) denotes receiving an offer and \(Z=0\) denotes not receiving one. Before the offer, the mean age is 52 years in the offer group and 50 years in the comparison group. Each group has \(n=200\) people and a sample standard deviation of 10 years. Calculate the difference, its standard error, and the standardized difference.

Difference in means. The estimated offer-group minus comparison-group difference is \(52-50=2\) years.

Standard error. For independent groups, the estimated standard error of the difference is \(SE=\sqrt{s_1^2/n_1+s_0^2/n_0}\). Substitution gives \(SE=\sqrt{10^2/200+10^2/200}=\sqrt{1}=1\) year. The test statistic for a zero difference is \(t=2/1=2.00\). With approximately 398 degrees of freedom, the two-sided p-value is about \(0.046\); the approximate 95% interval is \(2\pm1.96(1)=(0.04,3.96)\) years.

Standardized difference. The pooled standard deviation is \(\sqrt{(10^2+10^2)/2}=10\) years, so the standardized difference is \(2/10=0.200\). The observed baseline age difference is 2 years, or 0.200 pooled standard deviations. This is a reason to examine the assignment process and the pattern across baseline characteristics, not proof that the instrument is invalid. The p-value describes the data under a zero-difference null model; it is not the probability that the instrument is valid.

The uncertainty calculation above uses independent groups and the usual large-sample approximation for a difference in means. In an actual design, clustering, stratified assignment, or sampling weights may require a different standard error. The diagnostic should respect the assignment and sampling structure rather than automatically applying a simple two-group formula.

Placebo Outcomes

A placebo outcome is an outcome that, by the design and timing of the study, the instrument should not affect. A baseline outcome measured before assignment is one possible placebo. If the instrument predicts it, that pattern may suggest that assignment groups differed before treatment or that the instrument was related to pre-existing outcome determinants. The chosen placebo must genuinely precede the instrument or otherwise have no plausible causal route from the instrument; labeling an outcome “placebo” does not make that assumption true.

Worked Example: Checking a Pre-Assignment Outcome

In a hypothetical study, a binary instrument indicates whether a person received an offer. A health score recorded before the offer has mean 10.1 in the offer group and 10.0 in the comparison group. Each group contains 200 people, and each group’s sample standard deviation is 2.0 points. Calculate the difference, standard error, test statistic, and approximate 95% interval.

The difference is \(10.1-10.0=0.100\) health-score points. Under the independent-groups approximation, \(SE=\sqrt{2.0^2/200+2.0^2/200}=\sqrt{0.0400}=0.200\) points. Therefore, \(t=0.100/0.200=0.500\), with an approximate two-sided p-value of \(0.617\). The approximate 95% interval is \(0.100\pm1.96(0.200)=(-0.292,0.492)\) points.

These data do not show a statistically clear difference in this particular placebo outcome. The interval nevertheless includes differences in either direction, and this one check says nothing definitive about other baseline characteristics or unobserved determinants. It is supportive only in a limited sense; it does not establish the exclusion restriction for the study’s primary outcome.

Alternative Specifications as Sensitivity Checks

A specification check asks whether the substantive conclusion changes when reasonable modeling choices change. Choices might include a pre-specified set of baseline controls, a defensible functional form, or an alternative sample restriction tied to the design. The choices should be motivated by the assignment mechanism and stated transparently. Repeatedly searching specifications until one gives a preferred answer weakens, rather than strengthens, the evidential value of the analysis.

In a just-identified linear IV model, the estimate can be expressed as a reduced-form effect divided by a first-stage effect, with both quantities defined using the same specification. The following hypothetical calculations illustrate sensitivity to controls. They are diagnostic comparisons, not a proof that any specification satisfies exclusion.

Worked Example: Sensitivity to a Control Set

Suppose a hypothetical analysis reports the following reduced-form and first-stage coefficients for the same binary instrument. The reduced form is measured in outcome points per instrument change, and the first stage is measured in treatment units per instrument change. Under each just-identified specification, calculate the IV ratio.

Specification	Reduced form	First stage	IV ratio
No added controls	0.36	0.60	0.60
Pre-specified baseline controls	0.30	0.60	0.50
Alternative justified functional form	0.33	0.60	0.55

For the first specification, \(\hat{\beta}_{IV}=0.36/0.60=0.600\) outcome points per treatment unit. With baseline controls, \(\hat{\beta}_{IV}=0.30/0.60=0.500\) outcome points per treatment unit. Under the alternative functional form, \(\hat{\beta}_{IV}=0.33/0.60=0.550\) outcome points per treatment unit.

The estimates range from 0.500 to 0.600 outcome points per treatment unit. That variation is a reason to report the specifications and explain why they differ. It does not show which estimate is correct, and their similarity would not prove that the instrument has no direct effect. The ratio interpretation relies on the IV conditions for each specification; specification stability cannot replace an argument for those conditions.

Putting the Checks Together

A disciplined assessment separates the role of each source of evidence. Design records explain how instrument variation arose and help identify plausible violations. Balance and placebo checks search for observable patterns that would make the assignment account less credible. Alternative specifications reveal whether conclusions depend heavily on modeling decisions. Together, these checks can strengthen or weaken a case, but they do not observe the structural error \(u\) or rule out every direct path from \(Z\) to \(Y\).

Define the design claim.
State what instrument variation is supposed to represent, when it was generated, and why it should affect treatment.

Trace possible pathways.
Use institutional and timing information to consider whether the instrument could affect the outcome other than through the modeled treatment.

Run targeted diagnostics.
Examine predetermined covariates, genuinely unaffected placebo outcomes, and a small set of defensible alternative specifications.

Report limits as well as results.
Describe what each check can detect, what uncertainty remains, and why empirical compatibility is not proof of exclusion.

Common Mistakes and Why the Checks Matter

Treating a large p-value as proof of balance. A non-significant result may reflect limited precision. Report the estimated difference and its uncertainty, not just whether it crosses a threshold.
Equating balance on observed variables with exogeneity. Measured baseline characteristics do not exhaust unobserved determinants. Balance supports a design account but cannot establish it.
Calling a moment condition equivalent to conditional mean independence. The implication runs from \(E[u\mid Z,X]=E[u\mid X]\) to \(E[(Z-E[Z\mid X])u]=0\); the reverse does not generally hold.
Assuming a placebo is automatically valid. A placebo is informative only if the instrument could not plausibly affect it through the study’s timing or causal pathways.
Using specification stability as a validity test. Estimates can remain similar across specifications even if all share the same exclusion violation.

Key takeaway: Practical checks assess the credibility of the design through documented mechanisms and observable implications. They can reveal weaknesses and provide supporting evidence, but no pattern of balance, null placebo estimates, or specification stability proves the unobserved exclusion restriction.

Check Your Understanding

Answer each question using the distinctions made in this tutorial.

In the IV model, what roles do relevance and the exclusion and exogeneity restriction play?
Why does \(E[u\mid Z,X]=E[u\mid X]\) imply \(E[(Z-E[Z\mid X])u]=0\), and why is the converse not guaranteed?
A balance test has a large p-value but a wide confidence interval. What can and cannot be concluded?
What makes a pre-assignment outcome a useful placebo, and what would a detected association suggest?
Why does a similar IV estimate across several reasonable specifications not establish instrument validity?