Why discarded units matter
Matching can make treated and control units more comparable by selecting or weighting units with similar pretreatment covariates. But the resulting estimate need not describe the same population as the original study. If some units cannot be matched and are excluded, their covariate profiles and treatment effects no longer contribute to the estimate. The effect may therefore change its target population, even if the matching procedure improves balance among the units it retains.
The key question is not only whether the matched groups look alike. It is also: which units does the matched analysis represent? An estimate based on retained treated units generally concerns those treated units, while an estimate based on retained controls generally concerns those controls. If exclusions differ by treatment group or covariate profile, the target can differ from the full-sample ATE, ATT, or ATC.
Retention is a feature of the design and analysis, not a new potential outcome. In a treated-unit matching design, the retention rule may leave some treated units without suitable controls and exclude them; if all treated units are retained, the target is still the full-sample ATT. In a control-unit design, the analogous issue is whether some controls are excluded. Other designs may retain units from both groups. Thus “the matched effect” does not, by itself, name a unique population estimand.
How retention changes an estimand
For example, if the target is the treated units retained by a matching procedure, its population effect is
This is the average treatment effect among treated units that the design retains, not necessarily the ATT among all treated units. If instead the target is retained controls, the corresponding target is
A matched-sample target can be expressed more generally as an average of conditional treatment effects over the covariate distribution \(Q_M\) induced by the retained units and the matching weights:
Here \(Q_M\) is not automatically the original study's covariate distribution. It reflects which units were retained and how the matching procedure weighted them. In a finite matched sample, the analogous calculation averages effects over the represented units or matched sets using the analysis's specified weights. The precise target therefore requires stating both the retention rule and the weighting rule.
A target-population shift is distinct from confounding bias. A matching procedure can reduce differences in measured covariates among retained treated and control units and still change the population to which its estimate applies. Conversely, excluding units does not guarantee adequate balance or remove confounding. Target definition, balance, and causal identification are related but separate questions.
Assumptions for a causal interpretation
The population statement describes what an effect is about; assumptions are needed to identify that effect from observed outcomes. For a retained-treated target estimated by comparing treated units with matched controls, a standard set of assumptions is:
- Consistency: each unit's observed outcome equals its potential outcome under the treatment actually received, so \(Y_i=A_iY_i(1)+(1-A_i)Y_i(0)\).
- No interference: one unit's treatment does not change another unit's potential outcomes, and the treatment is sufficiently well-defined for the potential outcomes to be meaningful.
- Conditional exchangeability: given the measured pretreatment covariates used for adjustment, treatment assignment is independent of the potential outcomes. For a retained-treated ATT, the crucial counterfactual requirement is that treated and comparison units with the same relevant \(X\) have comparable \(Y(0)\).
- Relevant positivity: for covariate profiles represented among the retained treated units, there are control units with a positive probability of receiving control. The observed data must provide comparison information for the target profiles.
The matching rule should use pretreatment information and be specified without selecting units according to their observed outcomes. If exclusions are based on outcomes, the resulting selection can directly distort the outcome comparison. Even when retention depends only on treatment and covariates, the target is still the population selected by that rule, and the identification assumptions must hold for the comparisons that remain.
Under these assumptions, matching can support a causal estimate for the retained target when the retained comparison units adequately represent the missing potential outcomes for that target. The assumptions do not imply that the retained target equals the original population. Nor does covariate balance prove exchangeability: unmeasured confounding may remain.
Worked Example: Excluding Treated Units Changes the ATT
Consider a hypothetical population of 12 treated patients in two covariate groups. Assume the individual treatment effects are 2 outcome points for each low-risk patient and 10 points for each high-risk patient. A matching procedure retains 4 of the 8 low-risk treated patients and 1 of the 4 high-risk treated patients; the other treated patients are excluded because the matching design does not retain them. For this illustration, suppose the treatment effects are known, so the target averages can be calculated directly.
In the full treated population, 8 of 12 patients are low-risk and 4 are high-risk. Therefore the full ATT is
Among retained treated patients, the covariate mix is 4 low-risk and 1 high-risk, for 5 patients in total. Their average effect is
The retained treated group is 80% low-risk, compared with \(8/12=66.67\%\) in the full treated group. Because the high-risk group has the larger effect in this hypothetical example, the change in composition lowers the retained-group average by \(4.667-3.600=1.067\) outcome points. This difference reflects a different target population; it is not, by itself, evidence of bias in either target-specific average.
Three practical ways to see the shift
A useful audit begins with counts and covariate summaries for each treatment group, before and after matching. For each covariate, compare its distribution among all units in a group with its distribution among that group's retained units. Then describe the retained target in words. A statement such as “the estimate concerns treated patients retained by the matching design” is more informative than an unqualified label such as “the treatment effect.”
Worked Example: Control Exclusions and the ATC Target
In a hypothetical study, the full control group contains 10 patients: 6 with a low baseline severity score and 4 with a high score. Suppose the treatment effects are 3 outcome points for low-severity patients and 9 points for high-severity patients. A control-focused matching design retains 2 low-severity and 3 high-severity controls, excluding the remaining controls. Assume the listed effects are known for illustration.
The full-sample ATC is the average effect among all controls:
The retained control group contains 5 patients, of whom 3 are high-severity. Its target average is
The retained controls are 60% high-severity, while the full controls are 40% high-severity. The retained-control target consequently has a larger average effect in this example. Excluding controls changes the ATC only if it changes which controls, and therefore which effects, are represented; merely reducing the number of controls without changing their population mix need not change the target value.
The retained treated and retained control groups need not define the same covariate distribution. A matched contrast may use weights to make the groups represent a shared target, but that target is determined by the retained units and those weights. Do not infer the represented population merely from the fact that the data are described as “matched.”
Worked Example: A Matched Effect Can Differ From the Full ATE
Consider a hypothetical population with 20 low-risk and 10 high-risk units. Suppose the treatment effect is 2 points for every low-risk unit and 8 points for every high-risk unit. A matched analysis retains 8 low-risk and 2 high-risk units, and gives each retained unit equal weight. The full-population ATE is
The retained sample contains 10 units, 2 of them high-risk. Its equally weighted average effect is
The retained sample is 20% high-risk, compared with \(10/30=33.33\%\) in the full population. Since high-risk units have larger effects here, the retained average is lower by 0.800 points. In an actual observational study, both potential outcomes are not observed for any one unit, so this arithmetic illustrates the target shift rather than a directly available calculation. The matched outcome contrast estimates that target only under suitable identification assumptions.
Diagnostics, alternatives, and limitations
Report enough information for a reader to see what matching removed and what remained. At a minimum, give the numbers retained and discarded in each treatment group, retention proportions by group, and covariate summaries before and after matching. If a covariate strongly predicts outcomes or treatment and its distribution changes substantially after exclusions, that change is especially important for interpreting generalizability. A summary of overall retention can conceal very different retention rates across clinically or substantively important subgroups.
When the retained covariate distribution is markedly different from the original target, describe the matched estimate as applying to the retained target rather than implying it estimates the full-sample effect. If the scientific question concerns the original population, possible responses include changing the design or analysis to represent that target, or explicitly reporting a target-specific estimate alongside the limitations of extrapolating beyond the retained units. The choice depends on available data and the estimand of interest; retaining more units is not automatically preferable if the comparisons are poor.
The main alternatives differ in whose distribution is preserved. Treated-unit matching aims to represent treated units that can be matched and is naturally associated with an ATT-type target. Control-unit matching aims to represent retained controls and is naturally associated with an ATC-type target. A matched-sample effect describes the population induced by its retained units and weights. These labels are guides, not substitutes for specifying the actual target: exclusions and weights can modify it.
A common mistake is to say, “Matching removed only a few units, so the estimate still represents everyone.” The number discarded alone does not determine the size of a target shift. A small excluded subgroup can have unusual covariates or large effects; a larger excluded group can leave the average unchanged if its composition and effects preserve the same mean. Another mistake is to interpret a changed estimate as proof that matching corrected confounding. The change may instead arise because the estimate now averages effects over a different population. Compare retention and covariate composition, and state the target before interpreting the numerical difference.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A treated-unit match excludes high-severity treated patients but retains all low-severity treated patients. Which treated population does its ATT-type estimate directly target?
- In a hypothetical retained control group, 3 low-risk controls have effects of 2 points each and 2 high-risk controls have effects of 7 points each. Calculate the retained-control average effect.
- Why can discarding 5% of the study units change the target more than discarding 20%?
- Does close measured-covariate balance among retained matched units establish that the retained effect applies to the full study population? Explain.
- Suppose a matching analysis retains every treated unit but excludes some controls. Does this fact alone change the treated population that defines the ATT target?
Solutions. (1) It targets the treated patients retained by the design, not all treated patients; excluding high-severity patients changes the represented treated covariate mix. (2) The average is \([3(2)+2(7)]/5=(6+14)/5=4.000\) outcome points. (3) The 5% could be a highly distinctive subgroup with unusual covariates or treatment effects, while the 20% could have a composition and effects similar to those retained. The fraction alone does not determine the change in an average. (4) No. Balance concerns comparability of retained groups on measured covariates; it does not restore excluded units to the target or establish exchangeability for them. (5) Not by itself. If every treated unit remains in the target, the treated population defining the ATT is unchanged, though the available control comparisons, weighting, and causal identification may still be affected.