Why Matched Effects Are Easy to Misread
Matching can make treated and comparison units more alike on measured covariates, but the estimate still depends on which units were matched and how their outcomes were combined. A number calculated from matched data is not automatically the average treatment effect for all treated units, nor does a close match make the result universal. The central interpretive task is to identify the population represented by the matched contrast and to avoid extending the conclusion beyond it without justification.
This tutorial focuses on three recurring errors: labeling every matched estimate an ATT, overlooking units that matching discarded, and treating an effect for a restricted group as an effect for everyone. The introductory material on naming the estimand after matching distinguishes the target population from the estimator used to calculate an effect; here, that distinction is used to diagnose common interpretation errors.
For a simple one-to-one matched design in which each retained treated unit is paired with one comparison unit, the usual matched contrast is the average of the within-pair outcome differences. Its causal interpretation requires more than successful pairing. In particular, treatment and comparison outcomes must be consistently defined; one unit’s treatment must not affect another unit’s outcome; and, conditional on the measured covariates used to support the design, treatment assignment must be independent of potential outcomes. There must also be adequate overlap for the target units: comparable units must be available for the contrasts being made. Matching cannot verify the absence of unmeasured confounding.
These assumptions do not make every matched contrast target the same population. The retention rule, matching weights, and treatment-effect variation all matter. In the examples below, effects are expressed in outcome units and the arithmetic is shown explicitly. The data are hypothetical.
Mistake 1: Calling Every Matched Estimate an ATT
The ATT concerns all treated units in the defined population. A matched analysis may instead retain only treated units with suitable comparison units. If some treated units are excluded, the direct target of an equally weighted matched-pair estimate is typically the average effect for the retained treated units, not for all treated units. With matching ratios or weights other than one-to-one equal weighting, the precise target can differ again: units receiving more weight contribute more to the estimated average.
The correct label therefore cannot be chosen from the word “matched” alone. First identify which treated units contribute and how they are weighted. Then ask whether that set and weighting correspond to the treated population in the ATT definition.
Worked Example: A Caliper Retains Only Some Treated Units
Suppose a hypothetical study has four treated units. A caliper match retains treated units 1, 2, and 3 and finds one comparison unit for each; treated unit 4 has no acceptable match and is excluded. The observed outcome differences within the three pairs are 4, 3, and 5 points. Assume, for this calculation, that the matching assumptions support a causal interpretation for the retained pairs.
The equally weighted matched estimate is
This estimate summarizes the matched treated units, not automatically all four treated units. It would equal the ATT only under additional conditions—for example, if the excluded treated unit’s average effect were the same as the retained treated units’ average effect, or if the target population were explicitly restricted to the retained treated units. The observed matched differences alone do not establish either condition. Calling 4 points “the ATT” without addressing unit 4 overstates what this calculation directly targets.
Mistake 2: Treating Discarded Units as Irrelevant
A discarded unit does not contribute directly to a matched estimate, but it may matter to the difference between that estimate’s target and a broader target. Discarding treated units can change the represented treated population. Discarding comparison units can also reveal that some covariate profiles lack comparable observations. The number and characteristics of discarded units are therefore part of the interpretation, not merely a bookkeeping detail.
A useful way to see the issue is to partition the treated population into retained and discarded groups. If the retained fraction is \(r\), then the full ATT is a weighted average of the effect among retained treated units and the effect among discarded treated units:
This identity follows by conditioning the average effect among treated units on whether they were retained. It shows exactly why an estimate for retained treated units does not determine the full ATT: the discarded group’s effect and its share of all treated units also matter. The equality of the two effects is possible, but it should not be assumed simply because matching was performed.
Worked Example: How Excluded Treated Units Change the ATT
Consider a hypothetical population of 10 treated patients. Eight are retained by matching and have an average treatment effect of 2 points. Two high-risk patients are discarded for lack of comparable controls; suppose, for illustration, their average treatment effect is 10 points. The matched estimate targets the retained group’s effect, 2 points. The full ATT is
The 2-point matched effect and the 3.6-point full ATT are different because the discarded patients constitute 20% of the treated population and have a different average effect. In a real observational analysis, the discarded patients’ individual treatment effects cannot be observed directly; the values here are stipulated hypothetical values to show the arithmetic. The practical conclusion is not to “fill in” an unknown effect without evidence, but to report the restriction and avoid claiming that the matched estimate directly represents the full ATT.
A related diagnostic is to compare retained and discarded units on measured covariates and summarize how many units were excluded from each treatment group. Large differences can signal that the matched target is substantially narrower than the initial study population. Balance among retained pairs is useful, but it does not describe units that never entered those pairs.
Mistake 3: Treating a Restricted Effect as Universal
Even if the matched population is well defined and the within-population causal assumptions are plausible, its effect need not apply to people outside that population. Treatment effects may vary with age, baseline severity, or other characteristics. If matching retains mostly one covariate profile, the resulting average may be informative for that profile while differing from the average in a broader population.
The arithmetic of effect modification makes this clear. Suppose a population consists of two groups, with proportions \(w\) and \(1-w\) and average effects \(\tau_1\) and \(\tau_2\). The population effect is \(w\tau_1+(1-w)\tau_2\). A matched estimate concentrated in group 1 instead summarizes an effect near \(\tau_1\). It can be extended to the broader population only if the effects are sufficiently similar or if a valid method combines group-specific effects using the broader population’s composition.
Worked Example: A Matched Effect Does Not Describe Every Age Group
Suppose a hypothetical matched analysis retains 40 younger adults, whose average effect is 3 additional symptom-free days. Older adults are poorly represented in the matched sample. In the broader eligible population, assume for illustration that younger and older adults make up equal shares, and that their respective average effects are 3 days and 1 day.
The matched result describes the younger group: 3 days. The broader population average, under the stated equal shares, is
Thus, saying “the treatment adds 3 symptom-free days for the eligible population” would not follow from this matched analysis. That statement substitutes a restricted-group estimate for a broader average. The hypothetical population effect is 2 days, but a researcher could not obtain it from the younger-only matched estimate without information about the older group’s effect and the population shares. A difference between these effects is not a flaw in arithmetic; it reflects different target populations.
Checks, Alternatives, and Limits
Interpretation should be checked against the design actually used. Before describing a matched estimate, inspect the retained treated and comparison counts, the characteristics of excluded units, the covariate distributions in the matched sample, and the weights used to combine pairs. Balance diagnostics assess measured covariate similarity among the units included in the comparison; they do not demonstrate exchangeability, prove that no relevant units were discarded, or establish that an effect transports to a larger population.
When the matched target is too narrow for the research question, the remedy depends on the goal. One option is to define the estimand for the matched population and limit the conclusion to it. Another is to use a method that targets a broader population, such as standardization or weighting, if its assumptions and overlap requirements are defensible. A broader target is not achieved merely by relabeling a matched estimate, and alternative methods do not eliminate the need to justify causal assumptions. If some covariate profiles have no comparable treated and comparison units, changing methods may not supply the missing information.
Common errors can be summarized as follows:
- “It is matched, so it is the ATT.” Matching describes a design step, not the target by itself. Check which treated units contribute and how they are weighted.
- “The unmatched units do not matter.” They do not enter the matched arithmetic, but their exclusion can change the target and limit conclusions about the original treated population.
- “The matched effect applies to everyone.” That requires a defensible basis for generalization, including relevant effect similarity or valid standardization to the broader population.
- “Good balance proves the causal claim.” Balance is evidence about measured covariates in the analyzed sample. It cannot rule out unmeasured confounding or establish overlap for excluded units.
A concise interpretation should keep three questions separate: what contrast was computed, which population is represented by its weights, and what assumptions support a causal reading. If any answer is unclear, a strong causal or population-wide conclusion is premature.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A one-to-one match retains 60 of 75 treated units. Is the mean paired difference automatically the ATT for all 75 treated units? Explain.
- In a hypothetical population, 70% of treated units are retained, with an average effect of 2 points; the discarded 30% have an average effect of 7 points. Calculate the full ATT.
- A matched effect is estimated among patients with mild disease, while patients with severe disease are mostly excluded. What must be considered before applying the estimate to all patients?
- Why does good covariate balance among matched units not show that discarded units are irrelevant?
Solutions. (1) No. The paired difference directly represents the retained matched target under the relevant assumptions. It equals the full ATT only if the retained target and weighting match that estimand, or if additional conditions justify extending the result to the 15 excluded treated units. (2) The full ATT is \(0.70(2)+0.30(7)=1.4+2.1=3.5\) points. (3) Consider whether effects differ by disease severity, how many severe patients were excluded, and whether there is adequate overlap and evidence to standardize or otherwise generalize to them. The mild-disease estimate alone does not establish the all-patient effect. (4) Balance describes measured covariates among the units used in the matched comparison. It does not restore excluded units to the target or reveal their treatment effects; exclusions may alter the population represented by the estimate.