Comparing Matching And IV Estimates In One Study

One Treatment Question, Two Designs

Suppose a health service wants to know whether a supportive treatment increases patients’ symptom-free days during the 30 days after enrollment. An observational cohort can be analyzed by matching treated patients to similar untreated patients using measured covariates. If treatment receipt is also affected by a randomized offer, that offer may serve as an instrument for receipt. Both analyses address the same treatment question, but they do not automatically estimate the same causal effect.

The distinction is substantive, not merely computational. Matching relies on measured covariates being sufficient to control confounding and typically targets an effect for treated patients who can be matched. A valid instrumental-variables (IV) analysis can address unmeasured confounding of treatment receipt, but its estimate generally targets the effect among patients whose treatment receipt is changed by the instrument. Those populations may differ.

Definition: Let \(D\) indicate treatment receipt, with \(D=1\) for treated and \(D=0\) otherwise; let \(Y\) be the outcome; let \(X\) be measured pre-treatment covariates; and let \(Z\) be a binary instrument or encouragement. The matching analysis estimates an average treatment effect for a specified matched target group, often the average treatment effect among treated patients (ATT). Under the standard binary-instrument assumptions, the IV estimate is the local average treatment effect (LATE) among compliers: patients whose treatment receipt changes with \(Z\).

Worked Example: Matching the Observational Cohort

Worked Example: Estimating the Matched ATT

All numbers in this and the following examples are hypothetical. Suppose 100 treated patients are matched one-to-one, without replacement, to 100 untreated patients using a propensity score based on baseline severity, age, and prior health-care use. The outcome is symptom-free days in the 30-day period after enrollment.

Matched group	Number of patients	Mean symptom-free days
Treated	100	22
Untreated matches	100	18

The matched difference in means is

\widehat{\mathrm{ATT}}_{\mathrm{match}} = \bar{Y}_{D=1,\mathrm{matched}}-\bar{Y}_{D=0,\mathrm{matched}} =22-18=4\text{ days}.

Thus, for this matched sample, treated patients averaged four more symptom-free days than their matched untreated counterparts. This is a causal estimate of the ATT only if the matching design has achieved adequate comparability on all relevant confounders, in addition to the usual consistency and no-interference conditions. In particular, treatment assignment must be conditionally exchangeable given \(X\): after conditioning on measured covariates, potential outcomes must be independent of treatment. There must also be adequate treatment overlap for the target treated patients, so comparable untreated patients can be found.

In this hypothetical analysis, suppose the propensity-score distributions overlap for the retained patients and the absolute standardized mean difference for each prespecified baseline covariate is below 0.10 after matching. These checks support the claim that measured covariates are balanced and that the retained sample has usable overlap. They do not establish conditional exchangeability: an unmeasured prognostic factor could still differ between treatment groups.

Worked Example: The Instrumental-Variables Analysis

Now suppose an offer of the treatment is randomized. Some patients offered treatment do not receive it, while some not offered treatment obtain it through another route. The offer \(Z\) can be used as an instrument for actual receipt \(D\), provided the IV assumptions are credible.

Assumptions for a binary-instrument LATE: (1) Relevance: the instrument changes the probability of treatment receipt. (2) Independence: instrument assignment is independent of potential outcomes and potential treatment receipt, as would be expected under valid random assignment. (3) Exclusion: the instrument affects the outcome only through treatment receipt. (4) Monotonicity: there are no defiers—patients who would receive treatment when not offered but would refuse it when offered. Consistency and no interference are also needed to define the treatment and outcome being analyzed.

Worked Example: Calculating the Wald Estimate

In a hypothetical randomized encouragement study, 200 patients are offered treatment and 200 are not. Among those offered, 120 receive treatment; among those not offered, 80 receive it. Mean symptom-free days are 17.0 in the offer group and 15.9 in the no-offer group.

The first stage is the difference in treatment-receipt proportions:

\widehat{\Delta}_D =\frac{120}{200}-\frac{80}{200} =0.60-0.40=0.20.

The reduced-form effect of the offer on the outcome is

\widehat{\Delta}_Y=17.0-15.9=1.1\text{ days}.

The Wald estimate is the reduced form divided by the first stage:

\widehat{\tau}_{\mathrm{IV}} =\frac{\widehat{\Delta}_Y}{\widehat{\Delta}_D} =\frac{1.1\text{ days}}{0.20} =5.5\text{ days}.

Under the IV assumptions, the estimate is 5.5 additional symptom-free days per treatment receipt for compliers in the population represented by this encouragement comparison. It is not automatically the effect for every patient, nor the ATT among patients who chose treatment. The first stage is positive and sizeable in this sample. As a simple strength check, the standard error of the treatment-proportion difference under independent groups is \( \sqrt{0.60(0.40)/200+0.40(0.60)/200}\approx0.0490 \). The first-stage statistic is therefore about \(0.20/0.0490=4.08\), whose square is about 16.7 in this simple one-instrument setting. This is evidence that the offer predicts receipt, not evidence that exclusion or independence holds. In a real analysis, use an appropriate first-stage test for the study design and error structure.

Why the Estimates Can Differ

The Wald result follows from the binary-instrument assumptions. Under monotonicity, patients can be classified conceptually as compliers, always-takers, never-takers, or defiers; monotonicity rules out the last group. The instrument changes receipt only for compliers. Under independence and exclusion, the outcome difference between instrument groups is therefore the proportion of compliers multiplied by their average treatment effect. The treatment-receipt difference is that same proportion of compliers. Dividing the former by the latter isolates the complier average effect:

\frac{\text{outcome difference by }Z} {\text{treatment-receipt difference by }Z} = \frac{P(\text{complier})\,E[Y(1)-Y(0)\mid\text{complier}]} {P(\text{complier})} = E[Y(1)-Y(0)\mid\text{complier}].

Matching and IV can therefore differ even when both analyses are competently conducted. The matched ATT averages effects for treated patients in the matched target group. The IV LATE averages effects for compliers. The groups can have different covariate and effect distributions. In addition, matching depends on measured confounder adjustment, whereas IV depends on the instrument assumptions. Sampling variation and imperfect design implementation can also make estimates differ.

Worked Example: Different Target Populations, Different Effects

Consider a separate hypothetical illustration of effect heterogeneity. Suppose the matched treated group is 80% high-severity patients, whose treatment effect is 3 symptom-free days, and 20% lower-severity patients, whose treatment effect is 8 days. Its ATT is the weighted average

0.80(3)+0.20(8)=2.4+1.6=4.0\text{ days}.

Suppose instead that the IV compliers are 50% high-severity and 50% lower-severity, with the same severity-specific effects. Their LATE is

0.50(3)+0.50(8)=1.5+4.0=5.5\text{ days}.

Both averages are correct for their stated target populations. The two estimates differ by 1.5 days because the target groups have different severity compositions and the treatment effect varies by severity. The matching and IV calculations in this illustration are not required to yield the exact sample estimates in the earlier examples; the arithmetic shows how differing target populations can produce differing population effects.

Diagnostics, Alternatives, and Limitations

Report design-specific diagnostics alongside the point estimates. For matching, show covariate balance before and after matching, propensity-score overlap, the number of treated patients retained or discarded, and the matching rule. Examine important covariates individually as well as summary measures: an overall balance statistic can conceal a large imbalance in one prognostic variable. Poor overlap may require restricting the target population; it is not repaired merely by fitting a more elaborate propensity-score model.

For IV, report the first-stage estimate and an appropriate measure of its uncertainty or strength, along with the reduced-form estimate. Weak first stages can make IV estimates unstable and conventional inference unreliable. Randomization can support instrument independence, but exclusion is usually a substantive claim: an offer might change outcomes directly through counseling, monitoring, or access to other services. Monotonicity is also not directly established by the observed first-stage average. No balance table or strong first stage proves all IV assumptions.

The main alternative is not always to choose whichever estimate is larger or more statistically precise. Matching may be preferable when rich pre-treatment confounders are measured and credible overlap exists, but it cannot remove bias from unmeasured confounding. IV can address treatment confounding under its assumptions, but its local effect may not generalize beyond compliers, and exclusion can be difficult to defend. Presenting the estimates in parallel is informative when each design’s assumptions and target population are stated; treating them as interchangeable estimates of one universal effect is not.

A useful interpretation should consequently specify the outcome scale, target group, and design. For the hypothetical matching result, say “four additional symptom-free days on average for the matched treated patients, under conditional exchangeability and overlap.” For the IV result, say “5.5 additional days per treatment receipt for compliers, under instrument relevance, independence, exclusion, and monotonicity.” Neither statement should be shortened to an unqualified claim that treatment increases symptom-free days for all patients.

Common Mistakes

Comparing point estimates as if they target one parameter. The ATT and LATE can differ by definition when effects vary across people. First identify who contributes to each estimand.
Calling balance proof of no confounding. Good post-match balance concerns observed covariates only. It cannot rule out unmeasured confounding.
Treating a strong first stage as proof of a valid instrument. Relevance concerns prediction of treatment receipt; it does not establish exclusion, independence, or monotonicity.
Interpreting the Wald ratio as the effect of the offer. The offer’s reduced-form effect is 1.1 days in the example. Dividing by the 0.20 change in receipt yields a treatment effect for compliers, not the offer effect itself.
Assuming the larger estimate is more credible. Differences can arise from target-population composition, effect heterogeneity, sampling error, or violated assumptions. Magnitude alone cannot identify the reason.

Key takeaway: A parallel matching and IV analysis is most useful when it makes the design contrast explicit: matching targets a covariate-defined matched group under measured-confounding assumptions, while a valid binary-instrument analysis targets compliers under IV assumptions. Report each estimand, its diagnostics, and its limitations before comparing the numbers.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

A matched treated group averages 19 symptom-free days and its matched controls average 16. What is the estimated matched difference, and what target group does it describe?
An instrument changes treatment receipt from 0.30 to 0.50, and the outcome means differ by 0.8 days in the same direction. Calculate the Wald estimate.
Why does good post-match balance not establish that the matched estimate is unconfounded?
Name two assumptions, other than relevance, needed to interpret a binary-instrument Wald estimate as a LATE.
If the matched ATT is 4 days and the IV LATE is 5.5 days, does that discrepancy by itself show that one analysis is wrong? Explain.

Solutions. (1) The difference is \(19-16=3\) symptom-free days, for the treated patients represented by the matched target group; its causal interpretation requires the matching assumptions. (2) The first stage is \(0.50-0.30=0.20\), so the Wald estimate is \(0.8/0.20=4\) days per treatment receipt for compliers, if the IV assumptions hold. (3) Balance assesses recorded covariates, not unmeasured factors that may affect both treatment and outcome. (4) Independence, exclusion, and monotonicity are all required in addition to relevance; any two suffice as examples. (5) No. The estimands may concern different groups, effects may vary, and sampling error or assumption violations are also possible; the discrepancy calls for interpretation and diagnostic review, not automatic rejection of either estimate.