Negative-Control Outcomes for Instrument Diagnostics

Why Use an Outcome the Treatment Cannot Affect?

An instrumental-variable analysis depends on assumptions that are not generally settled by the observed association between an instrument and the primary outcome. A negative-control outcome provides a diagnostic: it is an outcome that the treatment under study cannot plausibly cause, but that may still reveal problems with the instrument or study design. The diagnostic is useful precisely because it asks whether the instrument is associated with something it should not affect under the relevant assumptions.

Let \(Z\) denote a binary instrument or encouragement, \(D\) the treatment received, and \(Y\) the primary outcome. Write \(W\) for a candidate negative-control outcome. For example, \(W\) might be a health measure recorded before an encouragement is assigned, or an outcome with a strong substantive reason to be unaffected by the treatment. Its role is not to estimate the effect of \(D\) on \(Y\). It is to help assess whether the observed instrument behaves as expected.

A useful candidate should be chosen for a clear causal reason, not simply because it happens to be available. If treatment could affect \(W\), then an association between \(Z\) and \(W\) could arise through the ordinary path \(Z\to D\to W\), and the diagnostic would no longer have its intended interpretation. Conversely, a negative control that is unrelated to the causes of the primary outcome may be a weak probe of instrument independence: it may fail to reveal the very imbalance that matters for \(Y\).

Why an Association Is a Warning Signal

The logic can be stated with potential outcomes. Let \(W(z,d)\) be the value of the negative-control outcome that would be observed under instrument level \(z\) and treatment level \(d\). Suppose, for this outcome, that treatment has no effect, so \(W(z,1)=W(z,0)\), and that the instrument has no direct effect on it, so \(W(1,d)=W(0,d)\). Together these say that \(W(z,d)\) is the same for every combination of \(z\) and \(d\); call this value \(W^*\). If instrument assignment is independent of \(W^*\), then the observed \(W\) has the same distribution across instrument groups.

The implication follows because consistency gives \(W=W^*\) for each person under the instrument and treatment actually received. Independence then gives the same distribution of \(W^*\) among those with \(Z=1\) and \(Z=0\), and therefore the same distribution of observed \(W\). For a binary \(W\), this entails equal proportions; for a continuous \(W\), it entails equal distributions, including equal means when those means exist.

This is a diagnostic implication, not a complete test of instrument validity. In particular, a difference in \(W\) does not by itself tell us whether the instrument is associated with relevant potential outcomes, has a direct effect on \(W\), or whether the supposed treatment-impossibility premise is wrong. Nor does it prove that the exclusion restriction for the primary outcome \(Y\) is violated. The instrument could affect \(W\) through a pathway that does not affect \(Y\), or an apparent difference could be due to chance.

Testing a Difference in a Binary Negative-Control Outcome

Suppose \(W\) is binary and the two instrument groups consist of independent observations. A simple diagnostic estimates the difference in negative-control outcome proportions, \(\hat{p}_1-\hat{p}_0\), where \(\hat{p}_z\) is the sample proportion with \(W=1\) in group \(Z=z\). To test \(H_0:p_1=p_0\), a large-sample two-proportion test uses the pooled estimate under the null. The test is appropriate when observations are independent and expected event and non-event counts in both groups are sufficiently large. For an interval estimating the difference, use the unpooled standard error.

A Null Result Is Not Validation

A diagnostic test can fail to detect an association even when the instrument groups differ meaningfully. This is especially likely when the sample is small, the negative-control outcome is uncommon, or the candidate outcome is only weakly related to the factors that threaten instrument independence. An interval is therefore more informative than a p-value alone: it shows which sizes of difference remain compatible with the data.

This example also illustrates why “the negative control was not significant” is not an adequate conclusion. The appropriate report describes the estimated contrast, its uncertainty, and the diagnostic's limitations. The test's p-value is the probability, under its null model, of a statistic at least as extreme as the observed one; it is not the probability that the instrument is valid.

Continuous Outcomes and the Source of a Signal

For a continuous negative-control outcome, one common diagnostic is the difference in group means. With independent groups, a Welch two-sample comparison estimates the standard error using each group's sample variance and sample size. Large samples make the normal approximation reasonable even if the outcome is not exactly normal; for small samples, distributional shape and the method used for inference require greater care. As with a binary outcome, the comparison detects a pattern, not its causal source.

Choosing and Reporting the Diagnostic

A negative-control outcome is most informative when its causal timing and relationship to the primary analysis are defensible. A pre-instrument outcome cannot be caused by a later treatment, which makes the treatment-impossibility premise especially clear. A post-instrument outcome may also be suitable, but only if substantive knowledge supports the claim that neither treatment nor a direct effect of the instrument can change it. If the instrument itself can affect the candidate outcome, an association may reveal a pathway specific to that outcome rather than instrument dependence relevant to \(Y\).

The following distinctions help keep conclusions calibrated:

If several negative-control outcomes are examined, report how many were considered and interpret the results with that multiplicity in mind. Looking across many tests increases the chance that at least one will appear unusual by chance. Likewise, a single well-chosen outcome is not a guarantee: different outcomes can be sensitive to different forms of imbalance or different instrument pathways.

The practical value of the diagnostic is therefore asymmetric. A strong association can expose a weakness in the assumptions or in the negative-control design and motivate investigation. A null result can be reassuring only in a limited sense, especially when the study has adequate precision and the negative control is a plausible proxy for relevant threats. It cannot establish the instrument's independence or exclusion for the primary outcome.

Common Mistake

Wrong reasoning: “The test of the negative-control outcome gave \(p=0.62\), so there is no association and the instrument assumptions are satisfied.” This treats failure to reject a null hypothesis as proof of equality, and then treats equality on one outcome as proof of broader causal assumptions.

Correction: A large p-value says the observed data are not unusual under the specified equal-distribution null; it does not show that the null is true. Examine the estimated contrast and interval, assess whether the candidate outcome truly cannot be affected by treatment or instrument, and ask whether it is informative about the threats relevant to the primary outcome. Even a precise null finding addresses only the diagnostic implication it was designed to probe.