Relevance as a Change in Treatment Receipt

What Relevance Measures

An instrument is useful only if treatment receipt differs across its values. For a binary instrument $Z$, coded 0 and 1, and a binary treatment receipt $D$, the basic relevance question is: how different is the probability of receiving treatment when $Z=1$ rather than $Z=0$? The answer is a probability difference. It is a contrast in treatment receipt, not a comparison of outcomes.

Definition: For a binary instrument $Z$ and binary treatment receipt $D$, define the population relevance contrast as $$ \Delta_D = P(D=1\mid Z=1)-P(D=1\mid Z=0). $$ The instrument is relevant in this probability-difference sense when $\Delta_D\ne 0$. A positive value means treatment receipt is more common among those with $Z=1$; a negative value means it is less common. The magnitude $|\Delta_D|$ describes the size of the difference in probability points.

For example, if the treatment probabilities are 0.60 and 0.45, the contrast is 0.15, or 15 percentage points. This does not mean treatment receipt increases by 15 percent: the difference is measured in percentage points, not as a relative percentage change. Nor does the definition say that changing $Z$ causes an individual to change treatment. It describes the difference between two conditional probabilities in the population.

The distinction between a probability contrast and a causal change matters. Let $D(0)$ and $D(1)$ denote the potential treatment receipts under the two instrument values, and let the observed receipt satisfy $D=D(Z)$. The observed contrast $\Delta_D$ compares people with $Z=1$ to people with $Z=0$. To identify it with the average causal change $E[D(1)-D(0)]$ requires additional assumptions about how $Z$ is assigned. Relevance by itself does not provide those assumptions.

Estimating the Contrast From Data

In a sample, calculate the treatment proportion separately for each instrument value. If $n_z$ individuals have $Z=z$, and $n_{1z}$ of them have $D=1$, then $\hat p_z=n_{1z}/n_z$. The sample relevance contrast is $\hat\Delta_D=\hat p_1-\hat p_0$. Its sign depends on which instrument value is coded 1; the coding must therefore be stated and checked.

\hat\Delta_D = \frac{\sum_{i:Z_i=1}D_i}{n_1} - \frac{\sum_{i:Z_i=0}D_i}{n_0} = \hat p_1-\hat p_0.

The calculation requires observations at both instrument values. If no one in the sample has $Z=0$, for example, the $Z=0$ treatment proportion cannot be calculated. Interpreting the sample contrast as an estimate for a broader population also requires that the data collection and measurement support that target: instrument and treatment values must be recorded consistently, and the observed units must provide a credible basis for estimating the relevant population probabilities.

Worked Example: Calculating a Positive Relevance Contrast

In a hypothetical sample, 300 people have $Z=1$, of whom 180 receive treatment. Another 300 have $Z=0$, of whom 90 receive treatment. Calculate and interpret the sample contrast.

For $Z=1$, $\hat p_1=180/300=0.600$. For $Z=0$, $\hat p_0=90/300=0.300$. Therefore,

\hat\Delta_D=0.600-0.300=0.300.

In this hypothetical sample, the treatment-receipt probability is 30 percentage points higher among people with $Z=1$ than among people with $Z=0$. This is a substantial observed difference. The calculation establishes the sample contrast, but by itself it does not establish that assigning $Z=1$ would cause an individual to receive treatment.

Both instrument groups are present, and the denominators match the stated group sizes, so both proportions can be calculated. No assignment assumption is needed to calculate this descriptive contrast. A causal interpretation would require additional assumptions not checked by these counts.

Why Compliance Types Clarify the Contrast

The four compliance types describe potential treatment receipts: always-takers have $D(0)=D(1)=1$, never-takers have $D(0)=D(1)=0$, compliers have $D(0)=0,D(1)=1$, and defiers have $D(0)=1,D(1)=0$. These definitions imply an exact identity for the potential-treatment contrast. They do not, by themselves, imply that the observed conditional probability contrast equals that identity.

D(1)-D(0)= \begin{cases} 1 & \text{for a complier},\\ -1 & \text{for a defier},\\ 0 & \text{for an always-taker or never-taker}. \end{cases}

Taking expectations gives a short derivation: each complier contributes one to the average change in potential treatment receipt, each defier contributes minus one, and the other two types contribute zero. Thus

E[D(1)-D(0)] = P(\text{complier})-P(\text{defier}).

This identity helps explain why the net potential-treatment change can be small even when some individuals respond to the instrument: increases among compliers can be offset by decreases among defiers. But it concerns potential receipts across instrument values. To use the observed difference $P(D=1\mid Z=1)-P(D=1\mid Z=0)$ as that average potential-treatment change requires instrument independence from the potential treatment receipts, along with consistency. A nonzero observed contrast is the relevance check; it is not proof of that independence.

Worked Example: A Negative Contrast and Instrument Coding

Suppose a hypothetical study records treatment for 120 people with $Z=1$, of whom 42 receive it, and 120 people with $Z=0$, of whom 66 receive it. Calculate the contrast using the stated coding.

The group proportions are $\hat p_1=42/120=0.350$ and $\hat p_0=66/120=0.550$. Hence,

\hat\Delta_D=0.350-0.550=-0.200.

Treatment receipt is 20 percentage points less common in the $Z=1$ group. This is a nonzero contrast, so it satisfies the descriptive definition of relevance, but its direction is negative. Reversing the coding of $Z$ would reverse the sign: the contrast would become $+0.200$, while its magnitude would remain $0.200$.

The data support the calculation because both groups have the same stated sample size and the event counts do not exceed those sizes. The negative sign is not a calculation error and does not, on its own, identify why the groups differ. It is especially important not to infer that individuals are defiers from a negative observed contrast; the observed group comparison is not a classification of individuals into compliance types.

Sampling Uncertainty and Diagnostics

A sample contrast is subject to sampling variation. Under independent sampling within the two instrument groups, a common large-sample standard error estimate is $\widehat{SE}(\hat\Delta_D)=\sqrt{\hat p_1(1-\hat p_1)/n_1+\hat p_0(1-\hat p_0)/n_0}$. A confidence interval based on this standard error indicates the range of population contrasts reasonably compatible with the sample under the approximation; it does not measure the probability that the instrument is valid.

Worked Example: A Small and Imprecise Contrast

In a hypothetical sample, 260 of 500 people with $Z=1$ receive treatment, while 245 of 500 with $Z=0$ receive treatment. Estimate the relevance contrast and use a large-sample 95% interval to assess its precision.

The proportions are $\hat p_1=260/500=0.520$ and $\hat p_0=245/500=0.490$, giving

\hat\Delta_D=0.520-0.490=0.030.

The estimated standard error is

\widehat{SE} = \sqrt{\frac{0.520(1-0.520)}{500} +\frac{0.490(1-0.490)}{500}} = \sqrt{0.000999} \approx 0.03161.

Using 1.96 standard errors on either side gives

0.030\pm 1.96(0.03161) = 0.030\pm 0.06196 \approx (-0.0320,\ 0.0920).

The sample estimate is a 3-percentage-point difference, but the interval is wide enough to include zero. Under the large-sample approximation and sampling assumptions, the data do not pin down the population contrast precisely. This is not proof that the population contrast equals zero; it is evidence that this sample does not distinguish a small negative contrast, no contrast, and a modest positive contrast well.

The approximation is reasonable as a basic illustration because each group has 500 observations and both treatment and non-treatment counts are large. With clustered, dependent, weighted, or otherwise complex data, the displayed standard error may not be appropriate; the uncertainty calculation must reflect the data design.

Several checks help detect problems before interpreting an estimate:

Check coding and denominators. Confirm which value is $Z=1$, that each $n_z$ counts all observations in that group, and that the treatment event is consistently coded as $D=1$.
Check group support. Very few observations at one instrument value make its treatment proportion unstable. A missing instrument group makes the contrast undefined in that sample.
Inspect magnitude and uncertainty together. A nonzero point estimate can be imprecise. A contrast near zero with a narrow interval and one near zero with a wide interval are different empirical situations.
Look for measurement or recording errors. Different definitions of treatment receipt across groups can create an apparent contrast that does not represent comparable treatment probabilities.
Do not use a universal cutoff. “Relevant” as a population definition means nonzero. In finite samples, practical strength depends on magnitude, uncertainty, sample size, and the inferential problem; a nonzero estimate alone does not guarantee useful precision.

What Relevance Does—and Does Not—Establish

For a binary $Z$, the unadjusted slope from a linear regression of $D$ on an intercept and $Z$ equals the difference in group means, $\hat p_1-\hat p_0$. The direct probability calculation is usually easiest to interpret because it displays both treatment rates and their difference. Regression can be useful in broader analyses, but a model-based or adjusted coefficient should not be confused with this unadjusted contrast: its interpretation depends on the model and adjustment choices.

Relevance is distinct from the other conditions needed for an instrumental-variable causal interpretation. It does not say that $Z$ is independent of potential outcomes or potential treatment receipts, and it does not say that $Z$ affects outcomes only through treatment. Those are separate assumptions. A large treatment-receipt difference cannot repair a failure of independence or exclusion; conversely, a plausible story about those assumptions cannot make a near-zero treatment contrast informative about treatment effects.

A common mistake is to reason, “Treatment is more frequent when $Z=1$, so $Z$ causes treatment and is a valid instrument.” The observed difference establishes only that treatment rates differ across the observed instrument groups. Whether that difference represents a causal shift, and whether the instrument supports the desired causal interpretation, requires additional assumptions. The correction is to report the contrast accurately, quantify its uncertainty, and assess the other assumptions separately.

Key takeaway: For a binary instrument, relevance is summarized by $P(D=1\mid Z=1)-P(D=1\mid Z=0)$. Calculate the two treatment probabilities before taking their difference, report its direction and size in percentage points, and keep the descriptive contrast separate from causal assumptions.

Check Your Understanding

Answer each question, then compare with the solutions below.

Among 200 people with $Z=1$, 130 receive treatment; among 250 with $Z=0$, 125 receive treatment. What is the sample relevance contrast, in percentage points?
If the treatment proportions are 0.42 for $Z=1$ and 0.57 for $Z=0$, is the contrast zero? What does its sign mean?
Why does a nonzero observed relevance contrast not by itself show that changing the instrument causes a change in treatment receipt?
Can a small estimated contrast with a confidence interval that includes zero establish that the population instrument is irrelevant? Explain.

Solutions. (1) $\hat p_1=130/200=0.650$ and $\hat p_0=125/250=0.500$, so $\hat\Delta_D=0.650-0.500=0.150$, or 15 percentage points. (2) The contrast is $0.42-0.57=-0.15$, so it is nonzero and negative: treatment receipt is 15 percentage points less common in the $Z=1$ group. (3) The contrast compares conditional probabilities among observed groups. A causal interpretation requires additional assumptions about instrument assignment and potential treatment receipt. (4) No. Such an interval indicates uncertainty compatible with zero and with some nonzero contrasts; it does not establish that the population contrast is exactly zero.